[
  {
    "path": ".gitignore",
    "content": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n.DS_Store\n.AppleDouble\n.LSOverride\n\n# C extensions\n*.so\n\n# Distribution / packaging\n.Python\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\neggs/\n.eggs/\nlib/\nlib64/\nparts/\nsdist/\nvar/\nwheels/\npip-wheel-metadata/\nshare/python-wheels/\n*.egg-info/\n.installed.cfg\n*.egg\nMANIFEST\n\n# PyInstaller\n#  Usually these files are written by a python script from a template\n#  before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# Unit test / coverage reports\nhtmlcov/\n.tox/\n.nox/\n.coverage\n.coverage.*\n.cache\nnosetests.xml\ncoverage.xml\n*.cover\n*.py,cover\n.hypothesis/\n.pytest_cache/\n\n# Translations\n*.mo\n*.pot\n\n# Django stuff:\n*.log\nlocal_settings.py\ndb.sqlite3\ndb.sqlite3-journal\n\n# Flask stuff:\ninstance/\n.webassets-cache\n\n# Scrapy stuff:\n.scrapy\n\n# Sphinx documentation\ndocs/_build/\n\n# PyBuilder\ntarget/\n\n# Jupyter Notebook\n.ipynb_checkpoints\n\n# IPython\nprofile_default/\nipython_config.py\n\n# pyenv\n.python-version\n\n# pipenv\n#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.\n#   However, in case of collaboration, if having platform-specific dependencies or dependencies\n#   having no cross-platform support, pipenv may install dependencies that don't work, or not\n#   install all needed dependencies.\n#Pipfile.lock\n\n# PEP 582; used by e.g. github.com/David-OConnor/pyflow\n__pypackages__/\n\n# Celery stuff\ncelerybeat-schedule\ncelerybeat.pid\n\n# SageMath parsed files\n*.sage.py\n\n# Environments\n.env\n.venv\nenv/\nvenv/\nENV/\nenv.bak/\nvenv.bak/\n\n# Spyder project settings\n.spyderproject\n.spyproject\n\n# Rope project settings\n.ropeproject\n\n# mkdocs documentation\n/site\n\n# mypy\n.mypy_cache/\n.dmypy.json\ndmypy.json\n\n# Pyre type checker\n.pyre/\n"
  },
  {
    "path": "README.md",
    "content": "# Claim-PT: Pretrained transformer framework on pediatric claims data for population specific tasks\n\nThis repository contains the tensorflow implementation of the following paper:\n\nPaper Name: Pretrained transformer framework on pediatric claims data for population specific tasks\n\nAuthors: Xianlong Zeng, Simon L. Linwood, Chang Liu\n\nAbstract: The adoption of electronic health records (EHR) has become universal during the past decade, which has afforded in-depth data-based research. By learning from the large amount of healthcare data, various data-driven models have been built to predict future events for different medical tasks, such as auto diagnosis and heart-attack prediction. Although EHR is abundant, the population that satisfies specific criteria for learning population-specific tasks is scarce, making it challenging to train data-hungry deep learning models. This study presents the Claim Pre-Training (Claim-PT) framework, a generic pre-training model that first trains on the entire pediatric claims dataset, followed by a discriminative fine-tuning on each population-specific task. The semantic meaning of medical events can be captured in the pre-training stage, and the effective knowledge transfer is completed through the task-aware fine-tuning stage. The fine-tuning process requires minimal parameter modification without changing the model architecture, which mitigates the data scarcity issue and helps train the deep learning model adequately on small patient cohorts. We conducted experiments on a real-world claims dataset with more than one million patient records. Experimental results on two downstream tasks demonstrated the effectiveness of our method: our general task-agnostic pre-training framework outperformed tailored task-specific models, achieving more than 10\\% higher in model performance as compared to baselines. In addition, our framework showed a great generalizability potential to transfer learned knowledge from one institution to another, paving the way for future healthcare model pre-training across institutions.\n\n[Paper Link](https://www.nature.com/articles/s41598-022-07545-1): Zeng, Xianlong, Simon L. Linwood, and Chang Liu. \"Pretrained transformer framework on pediatric claims data for population specific tasks.\" Scientific Reports 12, no. 1 (2022): 3651.\n\n# Environment\nUbuntu16.04, Python3.7, TensorFlow2.1\n\n\n# Model Pretraining\n\nin folder pretraining \n\n### List of hyper-parameter we used in the pre-training stage\n\n- MAX_VISIT=30\n- MAX_CODE=10\n- MAX_DEMO=2\n- PATIENT_DIM=100\n- BATCH_SIZE = 100\n- TRAIN_RATIO = 0.8\n- DATA_SIZE = len(age_seq)\n- EPOCHS = 1000\n\n# Downstream finetune\n\nin folder fineune\n\n# Attention playground\n\nin folder attention\n\n\n\n"
  },
  {
    "path": "attention/attention.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"regulation-chemistry\",\n   \"metadata\": {},\n   \"source\": [\n    \"### Case Study: Attention Visualization\\n\",\n    \"To illustrate the benefit of applying attention mechanisms and better understand how attention weights are changed from the pre-training stage to the fine-tuning stage, we analyze the learned attention weights at two different stages through a case study. \"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"id\": \"european-allowance\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stderr\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"wandb: WARNING W&B installed but not logged in.  Run `wandb login` or set the WANDB_API_KEY env variable.\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"import numpy as np\\n\",\n    \"import _pickle as pickle\\n\",\n    \"import pandas as pd\\n\",\n    \"import torch\\n\",\n    \"from bertviz import head_view\\n\",\n    \"from transformers import BertTokenizer, BertModel\\n\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"id\": \"moderate-translator\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"fake_data = pickle.load(open(\\\"./fake_data\\\", \\\"rb\\\"))\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"id\": \"juvenile-tennis\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"attention_weights_pretrain = pickle.load(open(\\\"./attention_weights_pretrain\\\", \\\"rb\\\"))\\n\",\n    \"attention_weights_finetune = pickle.load(open(\\\"./attention_weights_finetune\\\", \\\"rb\\\"))\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"id\": \"social-affairs\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"def attention_scores(weights, k):\\n\",\n    \"    weight = weights[1][k:k+2]\\n\",\n    \"    sentence = [list(x)[1][:7] for x in fake_data[k]]\\n\",\n    \"    \\n\",\n    \"    attn = torch.tensor(weight[:, :len(sentence), :len(sentence)])\\n\",\n    \"    attn = attn.repeat(1, 1, 1, 1)\\n\",\n    \"    attn_new = []\\n\",\n    \"    for _ in range(1):\\n\",\n    \"        attn_new.append(attn)\\n\",\n    \"    x = head_view(attn_new, sentence)\\n\",\n    \"    return \"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"floating-surge\",\n   \"metadata\": {},\n   \"source\": [\n    \"# Select Patient Index\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"id\": \"specialized-maximum\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"k=0\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"according-assets\",\n   \"metadata\": {},\n   \"source\": [\n    \"# Pretrain\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"id\": \"living-gates\",\n   \"metadata\": {\n    \"scrolled\": true\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<script src=\\\"https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.6/require.min.js\\\"></script>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    },\n    {\n     \"data\": {\n      \"text/html\": [\n       \"      \\n\",\n       \"        <div id='bertviz-973440206fb84c43a608ed88945dabed'>\\n\",\n       \"            <span style=\\\"user-select:none\\\">\\n\",\n       \"                Layer: <select id=\\\"layer\\\"></select>\\n\",\n       \"                \\n\",\n       \"            </span>\\n\",\n       \"            <div id='vis'></div>\\n\",\n       \"        </div>\\n\",\n       \"    \"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    },\n    {\n     \"data\": {\n      \"application/javascript\": \"/**\\n * @fileoverview Transformer Visualization D3 javascript code.\\n *\\n *\\n *  Based on: https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/visualization/attention.js\\n *\\n * Change log:\\n *\\n * 12/19/18  Jesse Vig   Assorted cleanup. Changed orientation of attention matrices.\\n * 12/29/20  Jesse Vig   Significant refactor.\\n * 12/31/20  Jesse Vig   Support multiple visualizations in single notebook.\\n * 02/06/21  Jesse Vig   Move require config from separate jupyter notebook step\\n * 05/03/21  Jesse Vig   Adjust height of visualization dynamically\\n * 07/25/21  Jesse Vig   Support layer filtering\\n **/\\n\\nrequire.config({\\n  paths: {\\n      d3: '//cdnjs.cloudflare.com/ajax/libs/d3/5.7.0/d3.min',\\n    jquery: '//ajax.googleapis.com/ajax/libs/jquery/2.0.0/jquery.min',\\n  }\\n});\\n\\nrequirejs(['jquery', 'd3'], function ($, d3) {\\n\\n    const params = {\\\"attention\\\": [{\\\"name\\\": null, \\\"attn\\\": [[[[0.16614454984664917, 0.1271992027759552, 0.11374139040708542, 0.15638020634651184, 0.20374387502670288, 0.11444535851478577, 0.11834541708230972], [0.16203011572360992, 0.1401018649339676, 0.13978800177574158, 0.1404246836900711, 0.13866281509399414, 0.13900551199913025, 0.1399870365858078], [0.14190000295639038, 0.14325059950351715, 0.14268504083156586, 0.14326979219913483, 0.14316092431545258, 0.14276131987571716, 0.14297233521938324], [0.1783653199672699, 0.13668516278266907, 0.13705065846443176, 0.13718663156032562, 0.1375586986541748, 0.1364724338054657, 0.13668116927146912], [0.17943505942821503, 0.13677170872688293, 0.13591942191123962, 0.13615824282169342, 0.13820093870162964, 0.13686884939670563, 0.13664573431015015], [0.1567888855934143, 0.1408628672361374, 0.14035286009311676, 0.1405779868364334, 0.14033928513526917, 0.14011918008327484, 0.14095890522003174], [0.1511572003364563, 0.1412300169467926, 0.14104074239730835, 0.14131779968738556, 0.14250311255455017, 0.1413128525018692, 0.14143821597099304]], [[0.047577470541000366, 0.04034224525094032, 0.03173127770423889, 0.03325849026441574, 0.05665835365653038, 0.040445178747177124, 0.03266838937997818], [0.03983418643474579, 0.03432993218302727, 0.0343342199921608, 0.034387726336717606, 0.034017566591501236, 0.03415762633085251, 0.03437581658363342], [0.0348617359995842, 0.03448709473013878, 0.034443072974681854, 0.034463874995708466, 0.0343620628118515, 0.03469187021255493, 0.03444177284836769], [0.03609755262732506, 0.03446199744939804, 0.03440416231751442, 0.034431908279657364, 0.03436928614974022, 0.034668270498514175, 0.0344182550907135], [0.04275275021791458, 0.034237828105688095, 0.03420986980199814, 0.03424009680747986, 0.03407548740506172, 0.03425997495651245, 0.03429330885410309], [0.04144877940416336, 0.034321531653404236, 0.034290462732315063, 0.03432182967662811, 0.03395794332027435, 0.03400801867246628, 0.0343194305896759], [0.03561968356370926, 0.034495096653699875, 0.034405313432216644, 0.03443482518196106, 0.034406013786792755, 0.03469153866171837, 0.03441842645406723]]]], \\\"left_text\\\": [\\\"P_99284\\\", \\\"D_948.0\\\", \\\"D_944.2\\\", \\\"M_POLYE\\\", \\\"P_272\\\", \\\"P_99254\\\", \\\"P_270\\\"], \\\"right_text\\\": [\\\"P_99284\\\", \\\"D_948.0\\\", \\\"D_944.2\\\", \\\"M_POLYE\\\", \\\"P_272\\\", \\\"P_99254\\\", \\\"P_270\\\"]}], \\\"default_filter\\\": \\\"0\\\", \\\"root_div_id\\\": \\\"bertviz-973440206fb84c43a608ed88945dabed\\\", \\\"layer\\\": null, \\\"heads\\\": null, \\\"include_layers\\\": [0]}; // HACK: {\\\"attention\\\": [{\\\"name\\\": null, \\\"attn\\\": [[[[0.16614454984664917, 0.1271992027759552, 0.11374139040708542, 0.15638020634651184, 0.20374387502670288, 0.11444535851478577, 0.11834541708230972], [0.16203011572360992, 0.1401018649339676, 0.13978800177574158, 0.1404246836900711, 0.13866281509399414, 0.13900551199913025, 0.1399870365858078], [0.14190000295639038, 0.14325059950351715, 0.14268504083156586, 0.14326979219913483, 0.14316092431545258, 0.14276131987571716, 0.14297233521938324], [0.1783653199672699, 0.13668516278266907, 0.13705065846443176, 0.13718663156032562, 0.1375586986541748, 0.1364724338054657, 0.13668116927146912], [0.17943505942821503, 0.13677170872688293, 0.13591942191123962, 0.13615824282169342, 0.13820093870162964, 0.13686884939670563, 0.13664573431015015], [0.1567888855934143, 0.1408628672361374, 0.14035286009311676, 0.1405779868364334, 0.14033928513526917, 0.14011918008327484, 0.14095890522003174], [0.1511572003364563, 0.1412300169467926, 0.14104074239730835, 0.14131779968738556, 0.14250311255455017, 0.1413128525018692, 0.14143821597099304]], [[0.047577470541000366, 0.04034224525094032, 0.03173127770423889, 0.03325849026441574, 0.05665835365653038, 0.040445178747177124, 0.03266838937997818], [0.03983418643474579, 0.03432993218302727, 0.0343342199921608, 0.034387726336717606, 0.034017566591501236, 0.03415762633085251, 0.03437581658363342], [0.0348617359995842, 0.03448709473013878, 0.034443072974681854, 0.034463874995708466, 0.0343620628118515, 0.03469187021255493, 0.03444177284836769], [0.03609755262732506, 0.03446199744939804, 0.03440416231751442, 0.034431908279657364, 0.03436928614974022, 0.034668270498514175, 0.0344182550907135], [0.04275275021791458, 0.034237828105688095, 0.03420986980199814, 0.03424009680747986, 0.03407548740506172, 0.03425997495651245, 0.03429330885410309], [0.04144877940416336, 0.034321531653404236, 0.034290462732315063, 0.03432182967662811, 0.03395794332027435, 0.03400801867246628, 0.0343194305896759], [0.03561968356370926, 0.034495096653699875, 0.034405313432216644, 0.03443482518196106, 0.034406013786792755, 0.03469153866171837, 0.03441842645406723]]]], \\\"left_text\\\": [\\\"P_99284\\\", \\\"D_948.0\\\", \\\"D_944.2\\\", \\\"M_POLYE\\\", \\\"P_272\\\", \\\"P_99254\\\", \\\"P_270\\\"], \\\"right_text\\\": [\\\"P_99284\\\", \\\"D_948.0\\\", \\\"D_944.2\\\", \\\"M_POLYE\\\", \\\"P_272\\\", \\\"P_99254\\\", \\\"P_270\\\"]}], \\\"default_filter\\\": \\\"0\\\", \\\"root_div_id\\\": \\\"bertviz-973440206fb84c43a608ed88945dabed\\\", \\\"layer\\\": null, \\\"heads\\\": null, \\\"include_layers\\\": [0]} is a template marker that is replaced by actual params.\\n    const TEXT_SIZE = 15;\\n    const BOXWIDTH = 110;\\n    const BOXHEIGHT = 22.5;\\n    const MATRIX_WIDTH = 115;\\n    const CHECKBOX_SIZE = 20;\\n    const TEXT_TOP = 30;\\n\\n    console.log(\\\"d3 version\\\", d3.version)\\n    let headColors;\\n    try {\\n        headColors = d3.scaleOrdinal(d3.schemeCategory10);\\n    } catch (err) {\\n        console.log('Older d3 version')\\n        headColors = d3.scale.category10();\\n    }\\n    let config = {};\\n    initialize();\\n    renderVis();\\n\\n    function initialize() {\\n        config.attention = params['attention'];\\n        config.filter = params['default_filter'];\\n        config.rootDivId = params['root_div_id'];\\n        config.nLayers = config.attention[config.filter]['attn'].length;\\n        config.nHeads = config.attention[config.filter]['attn'][0].length;\\n        config.layers = params['include_layers']\\n\\n        if (params['heads']) {\\n            config.headVis = new Array(config.nHeads).fill(false);\\n            params['heads'].forEach(x => config.headVis[x] = true);\\n        } else {\\n            config.headVis = new Array(config.nHeads).fill(true);\\n        }\\n        config.initialTextLength = config.attention[config.filter].right_text.length;\\n        config.layer_seq = (params['layer'] == null ? 0 : config.layers.findIndex(layer => params['layer'] === layer));\\n        config.layer = config.layers[config.layer_seq]\\n\\n        let layerEl = $(`#${config.rootDivId} #layer`);\\n        for (const layer of config.layers) {\\n            layerEl.append($(\\\"<option />\\\").val(layer).text(layer));\\n        }\\n        layerEl.val(config.layer).change();\\n        layerEl.on('change', function (e) {\\n            config.layer = +e.currentTarget.value;\\n            config.layer_seq = config.layers.findIndex(layer => config.layer === layer);\\n            renderVis();\\n        });\\n\\n        $(`#${config.rootDivId} #filter`).on('change', function (e) {\\n            config.filter = e.currentTarget.value;\\n            renderVis();\\n        });\\n    }\\n\\n    function renderVis() {\\n\\n        // Load parameters\\n        const attnData = config.attention[config.filter];\\n        const leftText = attnData.left_text;\\n        const rightText = attnData.right_text;\\n\\n        // Select attention for given layer\\n        const layerAttention = attnData.attn[config.layer_seq];\\n\\n        // Clear vis\\n        $(`#${config.rootDivId} #vis`).empty();\\n\\n        // Determine size of visualization\\n        const height = Math.max(leftText.length, rightText.length) * BOXHEIGHT + TEXT_TOP;\\n        const svg = d3.select(`#${config.rootDivId} #vis`)\\n            .append('svg')\\n            .attr(\\\"width\\\", \\\"100%\\\")\\n            .attr(\\\"height\\\", height + \\\"px\\\");\\n\\n        // Display tokens on left and right side of visualization\\n        renderText(svg, leftText, true, layerAttention, 0);\\n        renderText(svg, rightText, false, layerAttention, MATRIX_WIDTH + BOXWIDTH);\\n\\n        // Render attention arcs\\n        renderAttention(svg, layerAttention);\\n\\n        // Draw squares at top of visualization, one for each head\\n        drawCheckboxes(0, svg, layerAttention);\\n    }\\n\\n    function renderText(svg, text, isLeft, attention, leftPos) {\\n\\n        const textContainer = svg.append(\\\"svg:g\\\")\\n            .attr(\\\"id\\\", isLeft ? \\\"left\\\" : \\\"right\\\");\\n\\n        // Add attention highlights superimposed over words\\n        textContainer.append(\\\"g\\\")\\n            .classed(\\\"attentionBoxes\\\", true)\\n            .selectAll(\\\"g\\\")\\n            .data(attention)\\n            .enter()\\n            .append(\\\"g\\\")\\n            .attr(\\\"head-index\\\", (d, i) => i)\\n            .selectAll(\\\"rect\\\")\\n            .data(d => isLeft ? d : transpose(d)) // if right text, transpose attention to get right-to-left weights\\n            .enter()\\n            .append(\\\"rect\\\")\\n            .attr(\\\"x\\\", function () {\\n                var headIndex = +this.parentNode.getAttribute(\\\"head-index\\\");\\n                return leftPos + boxOffsets(headIndex);\\n            })\\n            .attr(\\\"y\\\", (+1) * BOXHEIGHT)\\n            .attr(\\\"width\\\", BOXWIDTH / activeHeads())\\n            .attr(\\\"height\\\", BOXHEIGHT)\\n            .attr(\\\"fill\\\", function () {\\n                return headColors(+this.parentNode.getAttribute(\\\"head-index\\\"))\\n            })\\n            .style(\\\"opacity\\\", 0.0);\\n\\n        const tokenContainer = textContainer.append(\\\"g\\\").selectAll(\\\"g\\\")\\n            .data(text)\\n            .enter()\\n            .append(\\\"g\\\");\\n\\n        // Add gray background that appears when hovering over text\\n        tokenContainer.append(\\\"rect\\\")\\n            .classed(\\\"background\\\", true)\\n            .style(\\\"opacity\\\", 0.0)\\n            .attr(\\\"fill\\\", \\\"lightgray\\\")\\n            .attr(\\\"x\\\", leftPos)\\n            .attr(\\\"y\\\", (d, i) => TEXT_TOP + i * BOXHEIGHT)\\n            .attr(\\\"width\\\", BOXWIDTH)\\n            .attr(\\\"height\\\", BOXHEIGHT);\\n\\n        // Add token text\\n        const textEl = tokenContainer.append(\\\"text\\\")\\n            .text(d => d)\\n            .attr(\\\"font-size\\\", TEXT_SIZE + \\\"px\\\")\\n            .style(\\\"cursor\\\", \\\"default\\\")\\n            .style(\\\"-webkit-user-select\\\", \\\"none\\\")\\n            .attr(\\\"x\\\", leftPos)\\n            .attr(\\\"y\\\", (d, i) => TEXT_TOP + i * BOXHEIGHT);\\n\\n        if (isLeft) {\\n            textEl.style(\\\"text-anchor\\\", \\\"end\\\")\\n                .attr(\\\"dx\\\", BOXWIDTH - 0.5 * TEXT_SIZE)\\n                .attr(\\\"dy\\\", TEXT_SIZE);\\n        } else {\\n            textEl.style(\\\"text-anchor\\\", \\\"start\\\")\\n                .attr(\\\"dx\\\", +0.5 * TEXT_SIZE)\\n                .attr(\\\"dy\\\", TEXT_SIZE);\\n        }\\n\\n        tokenContainer.on(\\\"mouseover\\\", function (d, index) {\\n\\n            // Show gray background for moused-over token\\n            textContainer.selectAll(\\\".background\\\")\\n                .style(\\\"opacity\\\", (d, i) => i === index ? 1.0 : 0.0)\\n\\n            // Reset visibility attribute for any previously highlighted attention arcs\\n            svg.select(\\\"#attention\\\")\\n                .selectAll(\\\"line[visibility='visible']\\\")\\n                .attr(\\\"visibility\\\", null)\\n\\n            // Hide group containing attention arcs\\n            svg.select(\\\"#attention\\\").attr(\\\"visibility\\\", \\\"hidden\\\");\\n\\n            // Set to visible appropriate attention arcs to be highlighted\\n            if (isLeft) {\\n                svg.select(\\\"#attention\\\").selectAll(\\\"line[left-token-index='\\\" + index + \\\"']\\\").attr(\\\"visibility\\\", \\\"visible\\\");\\n            } else {\\n                svg.select(\\\"#attention\\\").selectAll(\\\"line[right-token-index='\\\" + index + \\\"']\\\").attr(\\\"visibility\\\", \\\"visible\\\");\\n            }\\n\\n            // Update color boxes superimposed over tokens\\n            const id = isLeft ? \\\"right\\\" : \\\"left\\\";\\n            const leftPos = isLeft ? MATRIX_WIDTH + BOXWIDTH : 0;\\n            svg.select(\\\"#\\\" + id)\\n                .selectAll(\\\".attentionBoxes\\\")\\n                .selectAll(\\\"g\\\")\\n                .attr(\\\"head-index\\\", (d, i) => i)\\n                .selectAll(\\\"rect\\\")\\n                .attr(\\\"x\\\", function () {\\n                    const headIndex = +this.parentNode.getAttribute(\\\"head-index\\\");\\n                    return leftPos + boxOffsets(headIndex);\\n                })\\n                .attr(\\\"y\\\", (d, i) => TEXT_TOP + i * BOXHEIGHT)\\n                .attr(\\\"width\\\", BOXWIDTH / activeHeads())\\n                .attr(\\\"height\\\", BOXHEIGHT)\\n                .style(\\\"opacity\\\", function (d) {\\n                    const headIndex = +this.parentNode.getAttribute(\\\"head-index\\\");\\n                    if (config.headVis[headIndex])\\n                        if (d) {\\n                            return d[index];\\n                        } else {\\n                            return 0.0;\\n                        }\\n                    else\\n                        return 0.0;\\n                });\\n        });\\n\\n        textContainer.on(\\\"mouseleave\\\", function () {\\n\\n            // Unhighlight selected token\\n            d3.select(this).selectAll(\\\".background\\\")\\n                .style(\\\"opacity\\\", 0.0);\\n\\n            // Reset visibility attributes for previously selected lines\\n            svg.select(\\\"#attention\\\")\\n                .selectAll(\\\"line[visibility='visible']\\\")\\n                .attr(\\\"visibility\\\", null) ;\\n            svg.select(\\\"#attention\\\").attr(\\\"visibility\\\", \\\"visible\\\");\\n\\n            // Reset highlights superimposed over tokens\\n            svg.selectAll(\\\".attentionBoxes\\\")\\n                .selectAll(\\\"g\\\")\\n                .selectAll(\\\"rect\\\")\\n                .style(\\\"opacity\\\", 0.0);\\n        });\\n    }\\n\\n    function renderAttention(svg, attention) {\\n\\n        // Remove previous dom elements\\n        svg.select(\\\"#attention\\\").remove();\\n\\n        // Add new elements\\n        svg.append(\\\"g\\\")\\n            .attr(\\\"id\\\", \\\"attention\\\") // Container for all attention arcs\\n            .selectAll(\\\".headAttention\\\")\\n            .data(attention)\\n            .enter()\\n            .append(\\\"g\\\")\\n            .classed(\\\"headAttention\\\", true) // Group attention arcs by head\\n            .attr(\\\"head-index\\\", (d, i) => i)\\n            .selectAll(\\\".tokenAttention\\\")\\n            .data(d => d)\\n            .enter()\\n            .append(\\\"g\\\")\\n            .classed(\\\"tokenAttention\\\", true) // Group attention arcs by left token\\n            .attr(\\\"left-token-index\\\", (d, i) => i)\\n            .selectAll(\\\"line\\\")\\n            .data(d => d)\\n            .enter()\\n            .append(\\\"line\\\")\\n            .attr(\\\"x1\\\", BOXWIDTH)\\n            .attr(\\\"y1\\\", function () {\\n                const leftTokenIndex = +this.parentNode.getAttribute(\\\"left-token-index\\\")\\n                return TEXT_TOP + leftTokenIndex * BOXHEIGHT + (BOXHEIGHT / 2)\\n            })\\n            .attr(\\\"x2\\\", BOXWIDTH + MATRIX_WIDTH)\\n            .attr(\\\"y2\\\", (d, rightTokenIndex) => TEXT_TOP + rightTokenIndex * BOXHEIGHT + (BOXHEIGHT / 2))\\n            .attr(\\\"stroke-width\\\", 2)\\n            .attr(\\\"stroke\\\", function () {\\n                const headIndex = +this.parentNode.parentNode.getAttribute(\\\"head-index\\\");\\n                return headColors(headIndex)\\n            })\\n            .attr(\\\"left-token-index\\\", function () {\\n                return +this.parentNode.getAttribute(\\\"left-token-index\\\")\\n            })\\n            .attr(\\\"right-token-index\\\", (d, i) => i)\\n        ;\\n        updateAttention(svg)\\n    }\\n\\n    function updateAttention(svg) {\\n        svg.select(\\\"#attention\\\")\\n            .selectAll(\\\"line\\\")\\n            .attr(\\\"stroke-opacity\\\", function (d) {\\n                const headIndex = +this.parentNode.parentNode.getAttribute(\\\"head-index\\\");\\n                // If head is selected\\n                if (config.headVis[headIndex]) {\\n                    // Set opacity to attention weight divided by number of active heads\\n                    return d / activeHeads()\\n                } else {\\n                    return 0.0;\\n                }\\n            })\\n    }\\n\\n    function boxOffsets(i) {\\n        const numHeadsAbove = config.headVis.reduce(\\n            function (acc, val, cur) {\\n                return val && cur < i ? acc + 1 : acc;\\n            }, 0);\\n        return numHeadsAbove * (BOXWIDTH / activeHeads());\\n    }\\n\\n    function activeHeads() {\\n        return config.headVis.reduce(function (acc, val) {\\n            return val ? acc + 1 : acc;\\n        }, 0);\\n    }\\n\\n    function drawCheckboxes(top, svg) {\\n        const checkboxContainer = svg.append(\\\"g\\\");\\n        const checkbox = checkboxContainer.selectAll(\\\"rect\\\")\\n            .data(config.headVis)\\n            .enter()\\n            .append(\\\"rect\\\")\\n            .attr(\\\"fill\\\", (d, i) => headColors(i))\\n            .attr(\\\"x\\\", (d, i) => i * CHECKBOX_SIZE)\\n            .attr(\\\"y\\\", top)\\n            .attr(\\\"width\\\", CHECKBOX_SIZE)\\n            .attr(\\\"height\\\", CHECKBOX_SIZE);\\n\\n        function updateCheckboxes() {\\n            checkboxContainer.selectAll(\\\"rect\\\")\\n                .data(config.headVis)\\n                .attr(\\\"fill\\\", (d, i) => d ? headColors(i): lighten(headColors(i)));\\n        }\\n\\n        updateCheckboxes();\\n\\n        checkbox.on(\\\"click\\\", function (d, i) {\\n            if (config.headVis[i] && activeHeads() === 1) return;\\n            config.headVis[i] = !config.headVis[i];\\n            updateCheckboxes();\\n            updateAttention(svg);\\n        });\\n\\n        checkbox.on(\\\"dblclick\\\", function (d, i) {\\n            // If we double click on the only active head then reset\\n            if (config.headVis[i] && activeHeads() === 1) {\\n                config.headVis = new Array(config.nHeads).fill(true);\\n            } else {\\n                config.headVis = new Array(config.nHeads).fill(false);\\n                config.headVis[i] = true;\\n            }\\n            updateCheckboxes();\\n            updateAttention(svg);\\n        });\\n    }\\n\\n    function lighten(color) {\\n        const c = d3.hsl(color);\\n        const increment = (1 - c.l) * 0.6;\\n        c.l += increment;\\n        c.s -= increment;\\n        return c;\\n    }\\n\\n    function transpose(mat) {\\n        return mat[0].map(function (col, i) {\\n            return mat.map(function (row) {\\n                return row[i];\\n            });\\n        });\\n    }\\n\\n});\",\n      \"text/plain\": [\n       \"<IPython.core.display.Javascript object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"attention_scores(attention_weights_pretrain, k)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"statutory-meditation\",\n   \"metadata\": {},\n   \"source\": [\n    \"# Finetune\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 7,\n   \"id\": \"amino-burner\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<script src=\\\"https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.6/require.min.js\\\"></script>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    },\n    {\n     \"data\": {\n      \"text/html\": [\n       \"      \\n\",\n       \"        <div id='bertviz-651dd202c575406799ca073a08ef5e54'>\\n\",\n       \"            <span style=\\\"user-select:none\\\">\\n\",\n       \"                Layer: <select id=\\\"layer\\\"></select>\\n\",\n       \"                \\n\",\n       \"            </span>\\n\",\n       \"            <div id='vis'></div>\\n\",\n       \"        </div>\\n\",\n       \"    \"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    },\n    {\n     \"data\": {\n      \"application/javascript\": \"/**\\n * @fileoverview Transformer Visualization D3 javascript code.\\n *\\n *\\n *  Based on: https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/visualization/attention.js\\n *\\n * Change log:\\n *\\n * 12/19/18  Jesse Vig   Assorted cleanup. Changed orientation of attention matrices.\\n * 12/29/20  Jesse Vig   Significant refactor.\\n * 12/31/20  Jesse Vig   Support multiple visualizations in single notebook.\\n * 02/06/21  Jesse Vig   Move require config from separate jupyter notebook step\\n * 05/03/21  Jesse Vig   Adjust height of visualization dynamically\\n * 07/25/21  Jesse Vig   Support layer filtering\\n **/\\n\\nrequire.config({\\n  paths: {\\n      d3: '//cdnjs.cloudflare.com/ajax/libs/d3/5.7.0/d3.min',\\n    jquery: '//ajax.googleapis.com/ajax/libs/jquery/2.0.0/jquery.min',\\n  }\\n});\\n\\nrequirejs(['jquery', 'd3'], function ($, d3) {\\n\\n    const params = {\\\"attention\\\": [{\\\"name\\\": null, \\\"attn\\\": [[[[0.16445937752723694, 0.12579870223999023, 0.1133345440030098, 0.15633031725883484, 0.20919007062911987, 0.11322496086359024, 0.11766202002763748], [0.16152320802211761, 0.14000782370567322, 0.1398523598909378, 0.14041535556316376, 0.13919208943843842, 0.13906262814998627, 0.13994649052619934], [0.1411072164773941, 0.14333412051200867, 0.14292435348033905, 0.14334851503372192, 0.14325562119483948, 0.14297249913215637, 0.14305762946605682], [0.17711158096790314, 0.13652077317237854, 0.13727137446403503, 0.137272909283638, 0.1386304348707199, 0.1365974098443985, 0.13659557700157166], [0.17797741293907166, 0.13661958277225494, 0.13598830997943878, 0.13632971048355103, 0.13927258551120758, 0.1371464729309082, 0.13666586577892303], [0.15614137053489685, 0.14076903462409973, 0.14039446413516998, 0.14061200618743896, 0.1409381926059723, 0.14022590219974518, 0.14091895520687103], [0.15094415843486786, 0.1412247121334076, 0.14123418927192688, 0.14132803678512573, 0.14243265986442566, 0.14137880504131317, 0.1414574384689331]], [[0.04643920436501503, 0.03959910571575165, 0.031600456684827805, 0.03310973942279816, 0.05844629928469658, 0.03954439237713814, 0.03240245208144188], [0.039766352623701096, 0.03439924493432045, 0.034392524510622025, 0.034459188580513, 0.034056250005960464, 0.034218546003103256, 0.034431859850883484], [0.0349244587123394, 0.03453252092003822, 0.03443722426891327, 0.034467242658138275, 0.034302908927202225, 0.0347064845263958, 0.03444622829556465], [0.036057598888874054, 0.034504521638154984, 0.03441145271062851, 0.03444594889879227, 0.03432873636484146, 0.034707941114902496, 0.03443336486816406], [0.04230942204594612, 0.034347303211688995, 0.034300368279218674, 0.0343492291867733, 0.034209031611680984, 0.03433912247419357, 0.03436385095119476], [0.04137994721531868, 0.034371599555015564, 0.034335121512413025, 0.03437613323330879, 0.033850301057100296, 0.034061115235090256, 0.03437604010105133], [0.03564491868019104, 0.034539464861154556, 0.034410227090120316, 0.03444889560341835, 0.03437260538339615, 0.03471061959862709, 0.034431684762239456]]]], \\\"left_text\\\": [\\\"P_99284\\\", \\\"D_948.0\\\", \\\"D_944.2\\\", \\\"M_POLYE\\\", \\\"P_272\\\", \\\"P_99254\\\", \\\"P_270\\\"], \\\"right_text\\\": [\\\"P_99284\\\", \\\"D_948.0\\\", \\\"D_944.2\\\", \\\"M_POLYE\\\", \\\"P_272\\\", \\\"P_99254\\\", \\\"P_270\\\"]}], \\\"default_filter\\\": \\\"0\\\", \\\"root_div_id\\\": \\\"bertviz-651dd202c575406799ca073a08ef5e54\\\", \\\"layer\\\": null, \\\"heads\\\": null, \\\"include_layers\\\": [0]}; // HACK: {\\\"attention\\\": [{\\\"name\\\": null, \\\"attn\\\": [[[[0.16445937752723694, 0.12579870223999023, 0.1133345440030098, 0.15633031725883484, 0.20919007062911987, 0.11322496086359024, 0.11766202002763748], [0.16152320802211761, 0.14000782370567322, 0.1398523598909378, 0.14041535556316376, 0.13919208943843842, 0.13906262814998627, 0.13994649052619934], [0.1411072164773941, 0.14333412051200867, 0.14292435348033905, 0.14334851503372192, 0.14325562119483948, 0.14297249913215637, 0.14305762946605682], [0.17711158096790314, 0.13652077317237854, 0.13727137446403503, 0.137272909283638, 0.1386304348707199, 0.1365974098443985, 0.13659557700157166], [0.17797741293907166, 0.13661958277225494, 0.13598830997943878, 0.13632971048355103, 0.13927258551120758, 0.1371464729309082, 0.13666586577892303], [0.15614137053489685, 0.14076903462409973, 0.14039446413516998, 0.14061200618743896, 0.1409381926059723, 0.14022590219974518, 0.14091895520687103], [0.15094415843486786, 0.1412247121334076, 0.14123418927192688, 0.14132803678512573, 0.14243265986442566, 0.14137880504131317, 0.1414574384689331]], [[0.04643920436501503, 0.03959910571575165, 0.031600456684827805, 0.03310973942279816, 0.05844629928469658, 0.03954439237713814, 0.03240245208144188], [0.039766352623701096, 0.03439924493432045, 0.034392524510622025, 0.034459188580513, 0.034056250005960464, 0.034218546003103256, 0.034431859850883484], [0.0349244587123394, 0.03453252092003822, 0.03443722426891327, 0.034467242658138275, 0.034302908927202225, 0.0347064845263958, 0.03444622829556465], [0.036057598888874054, 0.034504521638154984, 0.03441145271062851, 0.03444594889879227, 0.03432873636484146, 0.034707941114902496, 0.03443336486816406], [0.04230942204594612, 0.034347303211688995, 0.034300368279218674, 0.0343492291867733, 0.034209031611680984, 0.03433912247419357, 0.03436385095119476], [0.04137994721531868, 0.034371599555015564, 0.034335121512413025, 0.03437613323330879, 0.033850301057100296, 0.034061115235090256, 0.03437604010105133], [0.03564491868019104, 0.034539464861154556, 0.034410227090120316, 0.03444889560341835, 0.03437260538339615, 0.03471061959862709, 0.034431684762239456]]]], \\\"left_text\\\": [\\\"P_99284\\\", \\\"D_948.0\\\", \\\"D_944.2\\\", \\\"M_POLYE\\\", \\\"P_272\\\", \\\"P_99254\\\", \\\"P_270\\\"], \\\"right_text\\\": [\\\"P_99284\\\", \\\"D_948.0\\\", \\\"D_944.2\\\", \\\"M_POLYE\\\", \\\"P_272\\\", \\\"P_99254\\\", \\\"P_270\\\"]}], \\\"default_filter\\\": \\\"0\\\", \\\"root_div_id\\\": \\\"bertviz-651dd202c575406799ca073a08ef5e54\\\", \\\"layer\\\": null, \\\"heads\\\": null, \\\"include_layers\\\": [0]} is a template marker that is replaced by actual params.\\n    const TEXT_SIZE = 15;\\n    const BOXWIDTH = 110;\\n    const BOXHEIGHT = 22.5;\\n    const MATRIX_WIDTH = 115;\\n    const CHECKBOX_SIZE = 20;\\n    const TEXT_TOP = 30;\\n\\n    console.log(\\\"d3 version\\\", d3.version)\\n    let headColors;\\n    try {\\n        headColors = d3.scaleOrdinal(d3.schemeCategory10);\\n    } catch (err) {\\n        console.log('Older d3 version')\\n        headColors = d3.scale.category10();\\n    }\\n    let config = {};\\n    initialize();\\n    renderVis();\\n\\n    function initialize() {\\n        config.attention = params['attention'];\\n        config.filter = params['default_filter'];\\n        config.rootDivId = params['root_div_id'];\\n        config.nLayers = config.attention[config.filter]['attn'].length;\\n        config.nHeads = config.attention[config.filter]['attn'][0].length;\\n        config.layers = params['include_layers']\\n\\n        if (params['heads']) {\\n            config.headVis = new Array(config.nHeads).fill(false);\\n            params['heads'].forEach(x => config.headVis[x] = true);\\n        } else {\\n            config.headVis = new Array(config.nHeads).fill(true);\\n        }\\n        config.initialTextLength = config.attention[config.filter].right_text.length;\\n        config.layer_seq = (params['layer'] == null ? 0 : config.layers.findIndex(layer => params['layer'] === layer));\\n        config.layer = config.layers[config.layer_seq]\\n\\n        let layerEl = $(`#${config.rootDivId} #layer`);\\n        for (const layer of config.layers) {\\n            layerEl.append($(\\\"<option />\\\").val(layer).text(layer));\\n        }\\n        layerEl.val(config.layer).change();\\n        layerEl.on('change', function (e) {\\n            config.layer = +e.currentTarget.value;\\n            config.layer_seq = config.layers.findIndex(layer => config.layer === layer);\\n            renderVis();\\n        });\\n\\n        $(`#${config.rootDivId} #filter`).on('change', function (e) {\\n            config.filter = e.currentTarget.value;\\n            renderVis();\\n        });\\n    }\\n\\n    function renderVis() {\\n\\n        // Load parameters\\n        const attnData = config.attention[config.filter];\\n        const leftText = attnData.left_text;\\n        const rightText = attnData.right_text;\\n\\n        // Select attention for given layer\\n        const layerAttention = attnData.attn[config.layer_seq];\\n\\n        // Clear vis\\n        $(`#${config.rootDivId} #vis`).empty();\\n\\n        // Determine size of visualization\\n        const height = Math.max(leftText.length, rightText.length) * BOXHEIGHT + TEXT_TOP;\\n        const svg = d3.select(`#${config.rootDivId} #vis`)\\n            .append('svg')\\n            .attr(\\\"width\\\", \\\"100%\\\")\\n            .attr(\\\"height\\\", height + \\\"px\\\");\\n\\n        // Display tokens on left and right side of visualization\\n        renderText(svg, leftText, true, layerAttention, 0);\\n        renderText(svg, rightText, false, layerAttention, MATRIX_WIDTH + BOXWIDTH);\\n\\n        // Render attention arcs\\n        renderAttention(svg, layerAttention);\\n\\n        // Draw squares at top of visualization, one for each head\\n        drawCheckboxes(0, svg, layerAttention);\\n    }\\n\\n    function renderText(svg, text, isLeft, attention, leftPos) {\\n\\n        const textContainer = svg.append(\\\"svg:g\\\")\\n            .attr(\\\"id\\\", isLeft ? \\\"left\\\" : \\\"right\\\");\\n\\n        // Add attention highlights superimposed over words\\n        textContainer.append(\\\"g\\\")\\n            .classed(\\\"attentionBoxes\\\", true)\\n            .selectAll(\\\"g\\\")\\n            .data(attention)\\n            .enter()\\n            .append(\\\"g\\\")\\n            .attr(\\\"head-index\\\", (d, i) => i)\\n            .selectAll(\\\"rect\\\")\\n            .data(d => isLeft ? d : transpose(d)) // if right text, transpose attention to get right-to-left weights\\n            .enter()\\n            .append(\\\"rect\\\")\\n            .attr(\\\"x\\\", function () {\\n                var headIndex = +this.parentNode.getAttribute(\\\"head-index\\\");\\n                return leftPos + boxOffsets(headIndex);\\n            })\\n            .attr(\\\"y\\\", (+1) * BOXHEIGHT)\\n            .attr(\\\"width\\\", BOXWIDTH / activeHeads())\\n            .attr(\\\"height\\\", BOXHEIGHT)\\n            .attr(\\\"fill\\\", function () {\\n                return headColors(+this.parentNode.getAttribute(\\\"head-index\\\"))\\n            })\\n            .style(\\\"opacity\\\", 0.0);\\n\\n        const tokenContainer = textContainer.append(\\\"g\\\").selectAll(\\\"g\\\")\\n            .data(text)\\n            .enter()\\n            .append(\\\"g\\\");\\n\\n        // Add gray background that appears when hovering over text\\n        tokenContainer.append(\\\"rect\\\")\\n            .classed(\\\"background\\\", true)\\n            .style(\\\"opacity\\\", 0.0)\\n            .attr(\\\"fill\\\", \\\"lightgray\\\")\\n            .attr(\\\"x\\\", leftPos)\\n            .attr(\\\"y\\\", (d, i) => TEXT_TOP + i * BOXHEIGHT)\\n            .attr(\\\"width\\\", BOXWIDTH)\\n            .attr(\\\"height\\\", BOXHEIGHT);\\n\\n        // Add token text\\n        const textEl = tokenContainer.append(\\\"text\\\")\\n            .text(d => d)\\n            .attr(\\\"font-size\\\", TEXT_SIZE + \\\"px\\\")\\n            .style(\\\"cursor\\\", \\\"default\\\")\\n            .style(\\\"-webkit-user-select\\\", \\\"none\\\")\\n            .attr(\\\"x\\\", leftPos)\\n            .attr(\\\"y\\\", (d, i) => TEXT_TOP + i * BOXHEIGHT);\\n\\n        if (isLeft) {\\n            textEl.style(\\\"text-anchor\\\", \\\"end\\\")\\n                .attr(\\\"dx\\\", BOXWIDTH - 0.5 * TEXT_SIZE)\\n                .attr(\\\"dy\\\", TEXT_SIZE);\\n        } else {\\n            textEl.style(\\\"text-anchor\\\", \\\"start\\\")\\n                .attr(\\\"dx\\\", +0.5 * TEXT_SIZE)\\n                .attr(\\\"dy\\\", TEXT_SIZE);\\n        }\\n\\n        tokenContainer.on(\\\"mouseover\\\", function (d, index) {\\n\\n            // Show gray background for moused-over token\\n            textContainer.selectAll(\\\".background\\\")\\n                .style(\\\"opacity\\\", (d, i) => i === index ? 1.0 : 0.0)\\n\\n            // Reset visibility attribute for any previously highlighted attention arcs\\n            svg.select(\\\"#attention\\\")\\n                .selectAll(\\\"line[visibility='visible']\\\")\\n                .attr(\\\"visibility\\\", null)\\n\\n            // Hide group containing attention arcs\\n            svg.select(\\\"#attention\\\").attr(\\\"visibility\\\", \\\"hidden\\\");\\n\\n            // Set to visible appropriate attention arcs to be highlighted\\n            if (isLeft) {\\n                svg.select(\\\"#attention\\\").selectAll(\\\"line[left-token-index='\\\" + index + \\\"']\\\").attr(\\\"visibility\\\", \\\"visible\\\");\\n            } else {\\n                svg.select(\\\"#attention\\\").selectAll(\\\"line[right-token-index='\\\" + index + \\\"']\\\").attr(\\\"visibility\\\", \\\"visible\\\");\\n            }\\n\\n            // Update color boxes superimposed over tokens\\n            const id = isLeft ? \\\"right\\\" : \\\"left\\\";\\n            const leftPos = isLeft ? MATRIX_WIDTH + BOXWIDTH : 0;\\n            svg.select(\\\"#\\\" + id)\\n                .selectAll(\\\".attentionBoxes\\\")\\n                .selectAll(\\\"g\\\")\\n                .attr(\\\"head-index\\\", (d, i) => i)\\n                .selectAll(\\\"rect\\\")\\n                .attr(\\\"x\\\", function () {\\n                    const headIndex = +this.parentNode.getAttribute(\\\"head-index\\\");\\n                    return leftPos + boxOffsets(headIndex);\\n                })\\n                .attr(\\\"y\\\", (d, i) => TEXT_TOP + i * BOXHEIGHT)\\n                .attr(\\\"width\\\", BOXWIDTH / activeHeads())\\n                .attr(\\\"height\\\", BOXHEIGHT)\\n                .style(\\\"opacity\\\", function (d) {\\n                    const headIndex = +this.parentNode.getAttribute(\\\"head-index\\\");\\n                    if (config.headVis[headIndex])\\n                        if (d) {\\n                            return d[index];\\n                        } else {\\n                            return 0.0;\\n                        }\\n                    else\\n                        return 0.0;\\n                });\\n        });\\n\\n        textContainer.on(\\\"mouseleave\\\", function () {\\n\\n            // Unhighlight selected token\\n            d3.select(this).selectAll(\\\".background\\\")\\n                .style(\\\"opacity\\\", 0.0);\\n\\n            // Reset visibility attributes for previously selected lines\\n            svg.select(\\\"#attention\\\")\\n                .selectAll(\\\"line[visibility='visible']\\\")\\n                .attr(\\\"visibility\\\", null) ;\\n            svg.select(\\\"#attention\\\").attr(\\\"visibility\\\", \\\"visible\\\");\\n\\n            // Reset highlights superimposed over tokens\\n            svg.selectAll(\\\".attentionBoxes\\\")\\n                .selectAll(\\\"g\\\")\\n                .selectAll(\\\"rect\\\")\\n                .style(\\\"opacity\\\", 0.0);\\n        });\\n    }\\n\\n    function renderAttention(svg, attention) {\\n\\n        // Remove previous dom elements\\n        svg.select(\\\"#attention\\\").remove();\\n\\n        // Add new elements\\n        svg.append(\\\"g\\\")\\n            .attr(\\\"id\\\", \\\"attention\\\") // Container for all attention arcs\\n            .selectAll(\\\".headAttention\\\")\\n            .data(attention)\\n            .enter()\\n            .append(\\\"g\\\")\\n            .classed(\\\"headAttention\\\", true) // Group attention arcs by head\\n            .attr(\\\"head-index\\\", (d, i) => i)\\n            .selectAll(\\\".tokenAttention\\\")\\n            .data(d => d)\\n            .enter()\\n            .append(\\\"g\\\")\\n            .classed(\\\"tokenAttention\\\", true) // Group attention arcs by left token\\n            .attr(\\\"left-token-index\\\", (d, i) => i)\\n            .selectAll(\\\"line\\\")\\n            .data(d => d)\\n            .enter()\\n            .append(\\\"line\\\")\\n            .attr(\\\"x1\\\", BOXWIDTH)\\n            .attr(\\\"y1\\\", function () {\\n                const leftTokenIndex = +this.parentNode.getAttribute(\\\"left-token-index\\\")\\n                return TEXT_TOP + leftTokenIndex * BOXHEIGHT + (BOXHEIGHT / 2)\\n            })\\n            .attr(\\\"x2\\\", BOXWIDTH + MATRIX_WIDTH)\\n            .attr(\\\"y2\\\", (d, rightTokenIndex) => TEXT_TOP + rightTokenIndex * BOXHEIGHT + (BOXHEIGHT / 2))\\n            .attr(\\\"stroke-width\\\", 2)\\n            .attr(\\\"stroke\\\", function () {\\n                const headIndex = +this.parentNode.parentNode.getAttribute(\\\"head-index\\\");\\n                return headColors(headIndex)\\n            })\\n            .attr(\\\"left-token-index\\\", function () {\\n                return +this.parentNode.getAttribute(\\\"left-token-index\\\")\\n            })\\n            .attr(\\\"right-token-index\\\", (d, i) => i)\\n        ;\\n        updateAttention(svg)\\n    }\\n\\n    function updateAttention(svg) {\\n        svg.select(\\\"#attention\\\")\\n            .selectAll(\\\"line\\\")\\n            .attr(\\\"stroke-opacity\\\", function (d) {\\n                const headIndex = +this.parentNode.parentNode.getAttribute(\\\"head-index\\\");\\n                // If head is selected\\n                if (config.headVis[headIndex]) {\\n                    // Set opacity to attention weight divided by number of active heads\\n                    return d / activeHeads()\\n                } else {\\n                    return 0.0;\\n                }\\n            })\\n    }\\n\\n    function boxOffsets(i) {\\n        const numHeadsAbove = config.headVis.reduce(\\n            function (acc, val, cur) {\\n                return val && cur < i ? acc + 1 : acc;\\n            }, 0);\\n        return numHeadsAbove * (BOXWIDTH / activeHeads());\\n    }\\n\\n    function activeHeads() {\\n        return config.headVis.reduce(function (acc, val) {\\n            return val ? acc + 1 : acc;\\n        }, 0);\\n    }\\n\\n    function drawCheckboxes(top, svg) {\\n        const checkboxContainer = svg.append(\\\"g\\\");\\n        const checkbox = checkboxContainer.selectAll(\\\"rect\\\")\\n            .data(config.headVis)\\n            .enter()\\n            .append(\\\"rect\\\")\\n            .attr(\\\"fill\\\", (d, i) => headColors(i))\\n            .attr(\\\"x\\\", (d, i) => i * CHECKBOX_SIZE)\\n            .attr(\\\"y\\\", top)\\n            .attr(\\\"width\\\", CHECKBOX_SIZE)\\n            .attr(\\\"height\\\", CHECKBOX_SIZE);\\n\\n        function updateCheckboxes() {\\n            checkboxContainer.selectAll(\\\"rect\\\")\\n                .data(config.headVis)\\n                .attr(\\\"fill\\\", (d, i) => d ? headColors(i): lighten(headColors(i)));\\n        }\\n\\n        updateCheckboxes();\\n\\n        checkbox.on(\\\"click\\\", function (d, i) {\\n            if (config.headVis[i] && activeHeads() === 1) return;\\n            config.headVis[i] = !config.headVis[i];\\n            updateCheckboxes();\\n            updateAttention(svg);\\n        });\\n\\n        checkbox.on(\\\"dblclick\\\", function (d, i) {\\n            // If we double click on the only active head then reset\\n            if (config.headVis[i] && activeHeads() === 1) {\\n                config.headVis = new Array(config.nHeads).fill(true);\\n            } else {\\n                config.headVis = new Array(config.nHeads).fill(false);\\n                config.headVis[i] = true;\\n            }\\n            updateCheckboxes();\\n            updateAttention(svg);\\n        });\\n    }\\n\\n    function lighten(color) {\\n        const c = d3.hsl(color);\\n        const increment = (1 - c.l) * 0.6;\\n        c.l += increment;\\n        c.s -= increment;\\n        return c;\\n    }\\n\\n    function transpose(mat) {\\n        return mat[0].map(function (col, i) {\\n            return mat.map(function (row) {\\n                return row[i];\\n            });\\n        });\\n    }\\n\\n});\",\n      \"text/plain\": [\n       \"<IPython.core.display.Javascript object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"attention_scores(attention_weights_finetune, k)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"personalized-intermediate\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  }\n ],\n \"metadata\": {\n  \"kernelspec\": {\n   \"display_name\": \"Python 3\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.7.11\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n"
  },
  {
    "path": "attention/get_att.py",
    "content": "import numpy as np\nimport tensorflow as tf\nfrom tensorflow.keras import layers\n\nclass DataGenerator(tf.keras.utils.Sequence):\n    def __init__(self, seqs, vocab_sizes, list_IDs, max_visit, max_code, batch_size=100, shuffle=True):\n        self.seqs = seqs\n        self.code_vocab = vocab_sizes[0]\n        self.cat_vocab = vocab_sizes[1]\n        self.list_IDs = list_IDs\n        self.max_visit = max_visit\n        self.max_code = max_code\n        self.batch_size = batch_size\n        self.shuffle = shuffle\n        self.on_epoch_end()\n\n    def __len__(self):\n        'Denotes the number of batches per epoch'\n        return int(np.ceil(len(self.list_IDs) / self.batch_size))\n\n    def __getitem__(self, index):\n        'Generate one batch of data'\n        indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]\n        list_IDs_temp = [self.list_IDs[k] for k in indexes]\n        X, y = self.__data_generation(list_IDs_temp)\n        return X, y\n\n    def on_epoch_end(self):\n        'Updates indexes after each epoch' \n        self.indexes = np.arange(len(self.list_IDs))\n        if self.shuffle == True:\n            np.random.shuffle(self.indexes)\n\n    def __data_generation(self, list_IDs_temp):\n        'Generates data containing batch_size samples' \n        demo_feature, code_feature, util_feature, date_feature, cls_feature = self.seqs\n        batch_demo, batch_code, batch_util, batch_date, batch_cls = [], [], [], [], []\n        for i, ID in enumerate(list_IDs_temp):\n            batch_demo.append(demo_feature[ID])\n            batch_code.append(code_feature[ID])\n            batch_util.append(util_feature[ID])\n            batch_date.append(date_feature[ID])\n            batch_cls.append(cls_feature[ID])\n        \n        batch_demo_feature = np.array(batch_demo)\n        batch_code_feature = self.code_padding(batch_code)\n        batch_util_feature = self.date_padding(batch_util)\n        batch_date_feature = self.date_padding(batch_date)\n        batch_cls = np.array(batch_cls)\n        \n        dic = (\n            {\n                'demo_feature': batch_demo_feature,\n                'code_feature': batch_code_feature,\n                'util_feature': batch_util_feature,\n                'date_feature': batch_date_feature,\n            },\n            {\n                'cls_label': batch_cls\n            })\n        return dic\n    \n    def date_padding(self, seq):\n        seq = [x[:-1] for x in seq]\n        \n        pad_seq = np.zeros((len(seq), self.max_visit))\n        for i, p in enumerate(seq):\n            pad_seq[i][:len(p)] = p[:self.max_visit]\n        return pad_seq\n    \n    def code_padding(self, seq):\n        seq = [x[:-1] for x in seq]\n        \n        X = np.zeros((len(seq), self.max_visit, self.max_code))\n        for i, p in enumerate(seq):\n            if len(p) > self.max_visit: \n                p = p[:self.max_visit]\n            for j, claim in enumerate(p):\n                claim = claim[:self.max_code]\n                X[i][j][:len(claim)] = claim\n        return X\n\ndef process_code(seq, vocab2int):\n    unseen = []\n    new_seq = []\n    for p in seq:\n        new_p = []\n        for v in p:\n            new_v = []\n            for c in v:\n                if c not in vocab2int: \n                    unseen.append(c)\n                    continue\n                    # vocab2int[c] = len(vocab2int)\n                new_v.append(vocab2int[c])\n            new_p.append(new_v)\n        new_seq.append(new_p)\n        \n    print(\"UNSEEN VOCAB:\",len(set(unseen)), len(unseen))\n    return new_seq\n\ndef process_util(seq, util2int):\n    new_seq = []\n    vocab2int = {\"PAD\":0,\"IP\":1,\"RX\":2,\"OP\":3}\n    for p in seq:\n        new_p = []\n        for v in p:\n            if \"IP\" in v:\n                new_v=1\n            elif \"RX\" in v:\n                new_v=2\n            else:\n                new_v=3\n            new_p.append(new_v)\n        new_seq.append(new_p)\n    return new_seq\n    \ndef process_demo(age_seq, sex_seq, vocab2int):\n    new_seq = []\n    for age, sex in zip(age_seq, sex_seq):\n        p = []\n        assert age in vocab2int\n        assert sex in vocab2int\n        \n        p.append(vocab2int[age])\n        p.append(vocab2int[sex])\n        new_seq.append(p)\n    return np.array(new_seq)\n\ndef get_cat(seq,code2cat):\n    new_seq = []\n    for p in seq:\n        new_p = []\n        for v in p:\n            new_v = []\n            for c in v:\n                new_c = code2cat[c]\n                new_v.append(new_c)\n            new_p.append(new_v)\n        new_seq.append(new_p)\n    return new_seq\n\n\nprint(\"==========LOADING DATA==========\")\nage_seq = pickle.load(open(\"../../suicideRisk/data/new_age_seq\",\"rb\"))\nsex_seq = pickle.load(open(\"../../suicideRisk/data/new_sex_seq\",\"rb\"))\n\nutil_seq = pickle.load(open(\"../../suicideRisk/data/new_util_seq\",\"rb\"))\ncode_seq = pickle.load(open(\"../../suicideRisk/data/new_code_seq\",\"rb\"))\ndate_seq = pickle.load(open(\"../../suicideRisk/data/new_date_seq\",\"rb\"))\nlabel_seq = pickle.load(open(\"../../suicideRisk/data/new_label_seq\",\"rb\"))\n\nprint(\"------LOADING DIC------\")\npath = \"/Users/xxz005/Desktop/RAW_DATA/code2cat/\"\n\ndiag2cat = pickle.load(open(path+\"diag2cat\",\"rb\"))\nproc2cat = pickle.load(open(path+\"proc2cat\",\"rb\"))\ndrug2cat = pickle.load(open(path+\"drug2cat\",\"rb\"))\n\ncode2cat = {**diag2cat, **proc2cat, **drug2cat}\n\ncode2int, util2int, demo2int, cat2int  = pickle.load(open(\"../../pretraining/model/vocabs/vocabs\",\"rb\"))\n\ncode_feature = process_code(code_seq, code2int)\nutil_feature = process_util(util_seq, util2int)\ndemo_feature = process_demo(age_seq, sex_seq, demo2int)\ndate_feature = date_seq\n\ncls_feature = np.array(label_seq).reshape((-1,1))\n\nMAX_VISIT=30\nMAX_CODE=10\nMAX_DEMO=2\nPATIENT_DIM=100\n\nBATCH_SIZE = 500\nTRAIN_RATIO = 0.7\nDATA_SIZE = len(age_seq)\nEPOCHS = 20\n\nparams = {\n    'seqs':[demo_feature, code_feature, util_feature, date_feature, cls_feature],\n    'vocab_sizes': [len(code2int), len(cat2int)],\n    'batch_size':100,\n    'max_visit':MAX_VISIT, \n    'max_code':MAX_CODE,\n}\n\n\ngenerator = DataGenerator(list_IDs=range(DATA_SIZE), shuffle=False, **params)\n\n\nmodel_path = \"./saveModel\"\n\nmodel = tf.keras.models.load_model(model_path)\nmodel_losses = {\n    \"cls_label\":tf.keras.losses.BinaryCrossentropy(),\n}\n\nmodel_metrics = {\n    \"cls_label\": tf.keras.metrics.AUC(),\n}\n\nopt = tf.keras.optimizers.Adam(learning_rate=0.0001)\nmodel.compile(optimizer=opt, loss=model_losses, metrics=model_metrics)\nprint(model.summary())\n\nlayer_name = 'multihead_attention-0'\nintermediate_layer_model = tf.keras.models.Model(inputs=model.input, outputs=model.get_layer(layer_name).output)\nencoded_features = intermediate_layer_model.predict(generator)\n\npickle.dump(encoded_features, open(\"attention_weights\", \"wb\"))\n"
  },
  {
    "path": "finetune/asthma/fine-tune.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"id\": \"wicked-finder\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 205 µs\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"%load_ext autotime\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"id\": \"random-fluid\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 780 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"import numpy as np\\n\",\n    \"import _pickle as pickle\\n\",\n    \"import pandas as pd\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"id\": \"diverse-vegetation\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"==========LOADING DATA==========\\n\",\n      \"time: 185 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(\\\"==========LOADING DATA==========\\\")\\n\",\n    \"age_seq = pickle.load(open(\\\"../data/new_age_seq\\\",\\\"rb\\\"))\\n\",\n    \"sex_seq = pickle.load(open(\\\"../data/new_sex_seq\\\",\\\"rb\\\"))\\n\",\n    \"        \\n\",\n    \"util_seq = pickle.load(open(\\\"../data/new_util_seq\\\",\\\"rb\\\"))\\n\",\n    \"code_seq = pickle.load(open(\\\"../data/new_code_seq\\\",\\\"rb\\\"))\\n\",\n    \"date_seq = pickle.load(open(\\\"../data/new_date_seq\\\",\\\"rb\\\"))\\n\",\n    \"label_seq = pickle.load(open(\\\"../data/new_label_seq\\\",\\\"rb\\\"))\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"id\": \"disabled-favor\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 3.28 s\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"import numpy as np\\n\",\n    \"import tensorflow as tf\\n\",\n    \"from tensorflow.keras import layers\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"id\": \"hollow-environment\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 5.82 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"class DataGenerator(tf.keras.utils.Sequence):\\n\",\n    \"    def __init__(self, seqs, vocab_sizes, list_IDs, max_visit, max_code, batch_size=100, shuffle=True):\\n\",\n    \"        self.seqs = seqs\\n\",\n    \"        self.code_vocab = vocab_sizes[0]\\n\",\n    \"        self.cat_vocab = vocab_sizes[1]\\n\",\n    \"        self.list_IDs = list_IDs\\n\",\n    \"        self.max_visit = max_visit\\n\",\n    \"        self.max_code = max_code\\n\",\n    \"        self.batch_size = batch_size\\n\",\n    \"        self.shuffle = shuffle\\n\",\n    \"        self.on_epoch_end()\\n\",\n    \"\\n\",\n    \"    def __len__(self):\\n\",\n    \"        'Denotes the number of batches per epoch'\\n\",\n    \"        return int(np.ceil(len(self.list_IDs) / self.batch_size))\\n\",\n    \"\\n\",\n    \"    def __getitem__(self, index):\\n\",\n    \"        'Generate one batch of data'\\n\",\n    \"        indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]\\n\",\n    \"        list_IDs_temp = [self.list_IDs[k] for k in indexes]\\n\",\n    \"        X, y = self.__data_generation(list_IDs_temp)\\n\",\n    \"        return X, y\\n\",\n    \"\\n\",\n    \"    def on_epoch_end(self):\\n\",\n    \"        'Updates indexes after each epoch' \\n\",\n    \"        self.indexes = np.arange(len(self.list_IDs))\\n\",\n    \"        if self.shuffle == True:\\n\",\n    \"            np.random.shuffle(self.indexes)\\n\",\n    \"\\n\",\n    \"    def __data_generation(self, list_IDs_temp):\\n\",\n    \"        'Generates data containing batch_size samples' \\n\",\n    \"        demo_feature, code_feature, util_feature, date_feature, cls_feature = self.seqs\\n\",\n    \"        batch_demo, batch_code, batch_util, batch_date, batch_cls = [], [], [], [], []\\n\",\n    \"        for i, ID in enumerate(list_IDs_temp):\\n\",\n    \"            batch_demo.append(demo_feature[ID])\\n\",\n    \"            batch_code.append(code_feature[ID])\\n\",\n    \"            batch_util.append(util_feature[ID])\\n\",\n    \"            batch_date.append(date_feature[ID])\\n\",\n    \"            batch_cls.append(cls_feature[ID])\\n\",\n    \"        \\n\",\n    \"        batch_demo_feature = np.array(batch_demo)\\n\",\n    \"        batch_code_feature = self.code_padding(batch_code)\\n\",\n    \"        batch_util_feature = self.date_padding(batch_util)\\n\",\n    \"        batch_date_feature = self.date_padding(batch_date)\\n\",\n    \"        batch_cls = np.array(batch_cls)\\n\",\n    \"        \\n\",\n    \"        dic = (\\n\",\n    \"            {\\n\",\n    \"                'demo_feature': batch_demo_feature,\\n\",\n    \"                'code_feature': batch_code_feature,\\n\",\n    \"                'util_feature': batch_util_feature,\\n\",\n    \"                'date_feature': batch_date_feature,\\n\",\n    \"            },\\n\",\n    \"            {\\n\",\n    \"                'cls_label': batch_cls\\n\",\n    \"            })\\n\",\n    \"        return dic\\n\",\n    \"    \\n\",\n    \"    def date_padding(self, seq):\\n\",\n    \"        seq = [x[:-1] for x in seq]\\n\",\n    \"        \\n\",\n    \"        pad_seq = np.zeros((len(seq), self.max_visit))\\n\",\n    \"        for i, p in enumerate(seq):\\n\",\n    \"            pad_seq[i][:len(p)] = p[:self.max_visit]\\n\",\n    \"        return pad_seq\\n\",\n    \"    \\n\",\n    \"    def code_padding(self, seq):\\n\",\n    \"        seq = [x[:-1] for x in seq]\\n\",\n    \"        \\n\",\n    \"        X = np.zeros((len(seq), self.max_visit, self.max_code))\\n\",\n    \"        for i, p in enumerate(seq):\\n\",\n    \"            if len(p) > self.max_visit: \\n\",\n    \"                p = p[:self.max_visit]\\n\",\n    \"            for j, claim in enumerate(p):\\n\",\n    \"                claim = claim[:self.max_code]\\n\",\n    \"                X[i][j][:len(claim)] = claim\\n\",\n    \"        return X\\n\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"id\": \"quick-webster\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 18 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"def create_code_mask(code_seq):\\n\",\n    \"    code_mask = tf.cast(tf.math.not_equal(code_seq, 0), tf.float32)\\n\",\n    \"    return code_mask[:,:,:,tf.newaxis]\\n\",\n    \"\\n\",\n    \"def create_visit_mask(seq):\\n\",\n    \"    visit_mask = tf.cast(tf.math.not_equal(seq, 0), tf.float32)\\n\",\n    \"    return visit_mask[:,:]\\n\",\n    \"\\n\",\n    \"def scaled_dot_product_attention(Q, K, V, Q_masks, K_masks):\\n\",\n    \"    d_k = K.get_shape().as_list()[-1] # d_model/h\\n\",\n    \"\\n\",\n    \"    outputs = tf.matmul(Q, tf.transpose(K, [0, 2, 1]))  # (h*N, T_q, T_k)\\n\",\n    \"    outputs /= d_k ** 0.5\\n\",\n    \"\\n\",\n    \"    padding_num = -1e+7\\n\",\n    \"    K_masks = tf.expand_dims(K_masks, 1) # (h*N, 1, T_k)\\n\",\n    \"    K_masks = tf.tile(K_masks, [1, tf.shape(Q)[1], 1]) # (h*N, T_q, T_k)\\n\",\n    \"    paddings = tf.ones_like(outputs) * padding_num\\n\",\n    \"    outputs = tf.where(tf.equal(K_masks, 0), paddings, outputs)  # (h*N, T_q, T_k)\\n\",\n    \"\\n\",\n    \"    outputs = tf.nn.softmax(outputs)\\n\",\n    \"    Q_masks = tf.expand_dims(Q_masks, -1) # (h*N, T_q, 1)\\n\",\n    \"    Q_masks = tf.tile(Q_masks, [1, 1, tf.shape(K)[1]]) # (h*N, T_q, T_k)\\n\",\n    \"    outputs = outputs * tf.cast(Q_masks, dtype=tf.float32)\\n\",\n    \"\\n\",\n    \"    return tf.matmul(outputs, V) # [h*N, T_q, d_model/h]\\n\",\n    \"\\n\",\n    \"class multihead_attention(tf.keras.layers.Layer):\\n\",\n    \"    def __init__(self, d_model, num_heads, name=\\\"multihead_attention\\\"):\\n\",\n    \"        super(multihead_attention, self).__init__(name=name)\\n\",\n    \"        self.num_heads = num_heads\\n\",\n    \"        self.d_model = d_model\\n\",\n    \"\\n\",\n    \"        assert d_model % self.num_heads == 0\\n\",\n    \"\\n\",\n    \"        self.query_dense = layers.Dense(units=d_model, use_bias=False)\\n\",\n    \"        self.key_dense = layers.Dense(units=d_model, use_bias=False)\\n\",\n    \"        self.value_dense = layers.Dense(units=d_model, use_bias=False)\\n\",\n    \"        self.add =layers.Add()\\n\",\n    \"        self.norm = layers.LayerNormalization()\\n\",\n    \"    \\n\",\n    \"    def call(self, queries, keys, values, query_masks, key_masks):\\n\",\n    \"        Q = self.query_dense(queries)\\n\",\n    \"        K = self.key_dense(keys)\\n\",\n    \"        V = self.value_dense(values)\\n\",\n    \"\\n\",\n    \"        # Split and concat\\n\",\n    \"        Q_ = tf.concat(tf.split(Q, self.num_heads, axis=2), axis=0) # (h*N, T_q, d_model/h)\\n\",\n    \"        K_ = tf.concat(tf.split(K, self.num_heads, axis=2), axis=0) # (h*N, T_k, d_model/h)\\n\",\n    \"        V_ = tf.concat(tf.split(V, self.num_heads, axis=2), axis=0) # (h*N, T_v, d_model/h)\\n\",\n    \"        query_masks = tf.tile(query_masks, [self.num_heads, 1]) # (h*N, T_q)\\n\",\n    \"        key_masks = tf.tile(key_masks, [self.num_heads, 1]) # (h*N, T_k)\\n\",\n    \"\\n\",\n    \"        # Attention\\n\",\n    \"        outputs = scaled_dot_product_attention(Q_, K_, V_, query_masks, key_masks) # (h*N, T_q, d_model/h)\\n\",\n    \"\\n\",\n    \"        # Restore shape\\n\",\n    \"        outputs = tf.concat(tf.split(outputs, self.num_heads, axis=0), axis=2) # (N, T_q, d_model)\\n\",\n    \"\\n\",\n    \"        # Residual connection\\n\",\n    \"        outputs = self.add([queries, outputs])\\n\",\n    \"        outputs = self.norm(outputs)\\n\",\n    \"        \\n\",\n    \"        return outputs\\n\",\n    \"\\n\",\n    \"class ffn(tf.keras.layers.Layer):\\n\",\n    \"    def __init__(self, d_model, ffn_dim, name=\\\"ffn\\\"):\\n\",\n    \"        super(ffn, self).__init__(name=name)\\n\",\n    \"        self.ffn_dim = ffn_dim\\n\",\n    \"        self.dense1 = layers.Dense(units=ffn_dim, activation=tf.nn.relu, use_bias=False)\\n\",\n    \"        self.dense2 = layers.Dense(units=d_model, use_bias=False)\\n\",\n    \"        self.add =layers.Add()\\n\",\n    \"        self.norm = layers.LayerNormalization()\\n\",\n    \"    \\n\",\n    \"    def call(self, inputs):\\n\",\n    \"        outputs = self.dense1(inputs)\\n\",\n    \"        outputs = self.dense2(outputs)\\n\",\n    \"        outputs = self.add([inputs, outputs])\\n\",\n    \"        outputs = self.norm(outputs)\\n\",\n    \"        return outputs\\n\",\n    \"\\n\",\n    \"def cat_recall(y_true, y_pred):\\n\",\n    \"    mask_value = tf.cast(tf.not_equal(tf.reduce_sum(y_true,axis=-1), 0), tf.float32)\\n\",\n    \"    true_positives = tf.cast(tf.reduce_sum(tf.multiply(tf.round(y_pred), y_true), axis=-1), tf.float32)\\n\",\n    \"    possible_positives = tf.cast(tf.reduce_sum(y_true, axis=-1), tf.float32)\\n\",\n    \"    values = true_positives / (possible_positives + 1e-7)\\n\",\n    \"    return tf.reduce_sum(values)/tf.reduce_sum(mask_value)\\n\",\n    \"\\n\",\n    \"def cat_loss_fun(y_true, y_pred):\\n\",\n    \"    loss = tf.cast(tf.keras.losses.BinaryCrossentropy(reduction='none')(y_true, y_pred), tf.float32)\\n\",\n    \"    mask = tf.cast(tf.not_equal(tf.reduce_sum(y_true,axis=-1), 0), tf.float32)\\n\",\n    \"    loss = tf.multiply(loss, mask)\\n\",\n    \"    # return tf.reduce_sum(loss)/tf.reduce_sum(mask)\\n\",\n    \"    return loss\\n\",\n    \"\\n\",\n    \"def model(\\n\",\n    \"    max_visit,\\n\",\n    \"    max_code,\\n\",\n    \"    max_demo,\\n\",\n    \"    \\n\",\n    \"    demo_vocab,\\n\",\n    \"    code_vocab,\\n\",\n    \"    date_vocab,\\n\",\n    \"    util_vocab,\\n\",\n    \"    cat_vocab,\\n\",\n    \"\\n\",\n    \"    patient_dim,\\n\",\n    \"    vocab_dim=100,\\n\",\n    \"    model_dim=100,\\n\",\n    \"    ffn_dim=100,\\n\",\n    \"    num_heads=2,\\n\",\n    \"    num_translayer=1,\\n\",\n    \"    \\n\",\n    \"    model_name=\\\"TransF\\\"):\\n\",\n    \"    \\n\",\n    \"    demo = layers.Input(shape=(max_demo, ), name=\\\"demo_feature\\\")  # max_demo = 2, age&sex\\n\",\n    \"    code_seq = layers.Input(shape=(max_visit, max_code), name=\\\"code_feature\\\") \\n\",\n    \"    util_seq = layers.Input(shape=(max_visit), name=\\\"util_feature\\\")\\n\",\n    \"    date_seq = layers.Input(shape=(max_visit), name=\\\"date_feature\\\")\\n\",\n    \"\\n\",\n    \"    inputs = [demo, code_seq, util_seq, date_seq]\\n\",\n    \"    \\n\",\n    \"    # demo embedding\\n\",\n    \"    demo_emb = layers.Embedding(input_dim=demo_vocab, output_dim=vocab_dim, mask_zero=True, name='demo_embedding')(demo)\\n\",\n    \"    demo_emb = layers.Lambda(lambda x: tf.keras.backend.sum(x, axis=1))(demo_emb)     \\n\",\n    \"\\n\",\n    \"    # code sequence\\n\",\n    \"    code_mask = layers.Lambda(create_code_mask)(code_seq)\\n\",\n    \"    code_emb = layers.Embedding(input_dim=code_vocab, \\n\",\n    \"                                output_dim=vocab_dim, \\n\",\n    \"                                name='code_embed')(code_seq)\\n\",\n    \"    code_emb = layers.Multiply()([code_emb, code_mask])\\n\",\n    \"    code_emb = tf.reduce_sum(code_emb, axis=2)     \\n\",\n    \"\\n\",\n    \"    \\n\",\n    \"    # visit mask\\n\",\n    \"    visit_mask = layers.Lambda(create_visit_mask)(date_seq)\\n\",\n    \"    \\n\",\n    \"    # util sequence   \\n\",\n    \"    util_emb = layers.Embedding(input_dim=util_vocab, output_dim=vocab_dim, mask_zero=True, name='util_embedding')(util_seq)\\n\",\n    \"    util_emb = layers.Multiply()([util_emb, visit_mask[:,:,tf.newaxis]])\\n\",\n    \"    \\n\",\n    \"    # date sequence    \\n\",\n    \"    date_emb = layers.Embedding(input_dim=date_vocab, output_dim=vocab_dim, mask_zero=True, name='date_embedding')(date_seq)\\n\",\n    \"    date_emb = layers.Multiply()([date_emb, visit_mask[:,:,tf.newaxis]])\\n\",\n    \"\\n\",\n    \"    # visit sequence\\n\",\n    \"    visit_emb = layers.Add()([code_emb, date_emb, util_emb]) \\n\",\n    \"\\n\",\n    \"    demo_emb = tf.expand_dims(demo_emb, 1) # (N, 1, emb_size)\\n\",\n    \"    demo_mask = tf.ones_like(tf.reduce_sum(demo_emb, axis=2), tf.float32) # (N, 1)\\n\",\n    \"    \\n\",\n    \"    for trans_layer in range(num_translayer):\\n\",\n    \"        multihead = multihead_attention(model_dim, num_heads, name=\\\"multihead_attention-\\\"+str(trans_layer))(\\n\",\n    \"            queries=tf.concat([demo_emb, visit_emb], 1),                         \\n\",\n    \"            keys=tf.concat([demo_emb, visit_emb], 1),\\n\",\n    \"            values=tf.concat([demo_emb, visit_emb], 1),\\n\",\n    \"            query_masks=tf.concat([demo_mask, visit_mask], 1),\\n\",\n    \"            key_masks=tf.concat([demo_mask, visit_mask], 1)\\n\",\n    \"        )\\n\",\n    \"        \\n\",\n    \"        visit_emb = ffn(model_dim, ffn_dim, name=\\\"ffn-\\\"+str(trans_layer))(multihead) # (N, max_visit, emb_size)\\n\",\n    \"            \\n\",\n    \"        demo_emb = visit_emb[:, :1, :] # (N, 1, emb_size)\\n\",\n    \"        visit_emb = visit_emb[:, 1:max_visit+1, :] # (N, max_visit, emb_size)\\n\",\n    \"        \\n\",\n    \"    patient_embedding = layers.Dense(patient_dim, activation=None, name=\\\"patient_embedding\\\")(tf.squeeze(demo_emb, [1]))\\n\",\n    \"    \\n\",\n    \"   \\n\",\n    \"    cls_label = layers.Dense(units=model_dim, activation=tf.nn.relu)(patient_embedding)\\n\",\n    \"    cls_label = layers.Dense(units=1, activation=tf.nn.sigmoid, name=\\\"cls_label\\\")(cls_label)\\n\",\n    \"        \\n\",\n    \" \\n\",\n    \"    outputs = [cls_label]\\n\",\n    \"\\n\",\n    \"    return tf.keras.Model(inputs=inputs, outputs=outputs, name=model_name)\\n\",\n    \"\\n\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 7,\n   \"id\": \"dependent-movie\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 4.81 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"def process_code(seq, vocab2int):\\n\",\n    \"    unseen = []\\n\",\n    \"    new_seq = []\\n\",\n    \"    for p in seq:\\n\",\n    \"        new_p = []\\n\",\n    \"        for v in p:\\n\",\n    \"            new_v = []\\n\",\n    \"            for c in v:\\n\",\n    \"                if c not in vocab2int: \\n\",\n    \"                    unseen.append(c)\\n\",\n    \"                    continue\\n\",\n    \"                    # vocab2int[c] = len(vocab2int)\\n\",\n    \"                new_v.append(vocab2int[c])\\n\",\n    \"            new_p.append(new_v)\\n\",\n    \"        new_seq.append(new_p)\\n\",\n    \"        \\n\",\n    \"    print(\\\"UNSEEN VOCAB:\\\",len(set(unseen)), len(unseen))\\n\",\n    \"    return new_seq\\n\",\n    \"\\n\",\n    \"def process_util(seq, util2int):\\n\",\n    \"    new_seq = []\\n\",\n    \"    vocab2int = {\\\"PAD\\\":0,\\\"IP\\\":1,\\\"RX\\\":2,\\\"OP\\\":3}\\n\",\n    \"    for p in seq:\\n\",\n    \"        new_p = []\\n\",\n    \"        for v in p:\\n\",\n    \"            if \\\"IP\\\" in v:\\n\",\n    \"                new_v=1\\n\",\n    \"            elif \\\"RX\\\" in v:\\n\",\n    \"                new_v=2\\n\",\n    \"            else:\\n\",\n    \"                new_v=3\\n\",\n    \"            new_p.append(new_v)\\n\",\n    \"        new_seq.append(new_p)\\n\",\n    \"    return new_seq\\n\",\n    \"    \\n\",\n    \"def process_demo(age_seq, sex_seq, vocab2int):\\n\",\n    \"    new_seq = []\\n\",\n    \"    for age, sex in zip(age_seq, sex_seq):\\n\",\n    \"        p = []\\n\",\n    \"        assert age in vocab2int\\n\",\n    \"        assert sex in vocab2int\\n\",\n    \"        \\n\",\n    \"        p.append(vocab2int[age])\\n\",\n    \"        p.append(vocab2int[sex])\\n\",\n    \"        new_seq.append(p)\\n\",\n    \"    return np.array(new_seq)\\n\",\n    \"\\n\",\n    \"def get_cat(seq,code2cat):\\n\",\n    \"    new_seq = []\\n\",\n    \"    for p in seq:\\n\",\n    \"        new_p = []\\n\",\n    \"        for v in p:\\n\",\n    \"            new_v = []\\n\",\n    \"            for c in v:\\n\",\n    \"                new_c = code2cat[c]\\n\",\n    \"                new_v.append(new_c)\\n\",\n    \"            new_p.append(new_v)\\n\",\n    \"        new_seq.append(new_p)\\n\",\n    \"    return new_seq\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 8,\n   \"id\": \"ongoing-manufacturer\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"------LOADING DIC------\\n\",\n      \"time: 69.2 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(\\\"------LOADING DIC------\\\")\\n\",\n    \"path = \\\"/Users/xxz005/Desktop/RAW_DATA/code2cat/\\\"\\n\",\n    \"\\n\",\n    \"diag2cat = pickle.load(open(path+\\\"diag2cat\\\",\\\"rb\\\"))\\n\",\n    \"proc2cat = pickle.load(open(path+\\\"proc2cat\\\",\\\"rb\\\"))\\n\",\n    \"drug2cat = pickle.load(open(path+\\\"drug2cat\\\",\\\"rb\\\"))\\n\",\n    \"\\n\",\n    \"code2cat = {**diag2cat, **proc2cat, **drug2cat}\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 9,\n   \"id\": \"wrong-warrior\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 36.3 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"code2int, util2int, demo2int, cat2int  = pickle.load(open(\\\"../../pretraining/model/vocabs/vocabs\\\",\\\"rb\\\"))\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 10,\n   \"id\": \"invalid-silicon\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"UNSEEN VOCAB: 3 4\\n\",\n      \"time: 120 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"code_feature = process_code(code_seq, code2int)\\n\",\n    \"util_feature = process_util(util_seq, util2int)\\n\",\n    \"demo_feature = process_demo(age_seq, sex_seq, demo2int)\\n\",\n    \"date_feature = date_seq\\n\",\n    \"\\n\",\n    \"cls_feature = np.array(label_seq).reshape((-1,1))\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 11,\n   \"id\": \"adverse-arkansas\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 1.92 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"MAX_VISIT=30\\n\",\n    \"MAX_CODE=10\\n\",\n    \"MAX_DEMO=2\\n\",\n    \"PATIENT_DIM=100\\n\",\n    \"\\n\",\n    \"BATCH_SIZE = 500\\n\",\n    \"TRAIN_RATIO = 0.5\\n\",\n    \"DATA_SIZE = len(age_seq)\\n\",\n    \"EPOCHS = 20\\n\",\n    \"\\n\",\n    \"params = {\\n\",\n    \"    'seqs':[demo_feature, code_feature, util_feature, date_feature, cls_feature],\\n\",\n    \"    'vocab_sizes': [len(code2int), len(cat2int)],\\n\",\n    \"    'batch_size':100,\\n\",\n    \"    'max_visit':MAX_VISIT, \\n\",\n    \"    'max_code':MAX_CODE,\\n\",\n    \"}\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 12,\n   \"id\": \"sorted-wholesale\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 933 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"from sklearn.model_selection import train_test_split\\n\",\n    \"train_IDs, valid_IDs = train_test_split(range(DATA_SIZE), train_size=TRAIN_RATIO, random_state=42)\\n\",\n    \"train_generator = DataGenerator(list_IDs=train_IDs, shuffle=True, **params)\\n\",\n    \"valid_generator = DataGenerator(list_IDs=valid_IDs, shuffle=False, **params)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"separated-layout\",\n   \"metadata\": {},\n   \"source\": [\n    \"# Pretrain\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 13,\n   \"id\": \"suspended-settlement\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 393 µs\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"# model_path = \\\"../../pretraining/model/saveModel\\\"\\n\",\n    \"\\n\",\n    \"# model = tf.keras.models.load_model(model_path)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 14,\n   \"id\": \"handy-impression\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stderr\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"/usr/local/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: cpt is not loaded, but a Lambda layer uses it. It may cause errors.\\n\",\n      \"  , UserWarning)\\n\"\n     ]\n    },\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 1.84 s\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"model_path = \\\"/Users/xxz005/Desktop/saveModel\\\"\\n\",\n    \"\\n\",\n    \"model = tf.keras.models.load_model(model_path)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 15,\n   \"id\": \"involved-genetics\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 8.37 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"model_losses = {\\n\",\n    \"    \\\"cls_label\\\":tf.keras.losses.BinaryCrossentropy(),\\n\",\n    \"}\\n\",\n    \"\\n\",\n    \"model_metrics = {\\n\",\n    \"    \\\"cls_label\\\": tf.keras.metrics.AUC(),\\n\",\n    \"}\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 16,\n   \"id\": \"veterinary-chinese\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 16.6 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"opt = tf.keras.optimizers.Adam(learning_rate=0.0001)\\n\",\n    \"model.compile(optimizer=opt, loss=model_losses, metrics=model_metrics)\\n\",\n    \"# print(model.summary())\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 17,\n   \"id\": \"vanilla-karen\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"9/9 [==============================] - 1s 26ms/step - loss: 5.8961 - cls_label_loss: 5.8961 - cls_label_auc: 0.4489\\n\"\n     ]\n    },\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"[5.853375434875488, 5.853375434875488, 0.4493659734725952]\"\n      ]\n     },\n     \"execution_count\": 17,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    },\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 946 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"model.evaluate(valid_generator)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 18,\n   \"id\": \"verbal-kingdom\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Epoch 1/20\\n\",\n      \"WARNING:tensorflow:Gradients do not exist for variables ['code_label/kernel:0', 'code_label/bias:0', 'cat_label/kernel:0', 'cat_label/bias:0'] when minimizing the loss.\\n\",\n      \"WARNING:tensorflow:Gradients do not exist for variables ['code_label/kernel:0', 'code_label/bias:0', 'cat_label/kernel:0', 'cat_label/bias:0'] when minimizing the loss.\\n\",\n      \"9/9 - 3s - loss: 2.6438 - cls_label_loss: 2.6438 - cls_label_auc: 0.5739 - val_loss: 1.8408 - val_cls_label_loss: 1.8408 - val_cls_label_auc: 0.5634\\n\",\n      \"Epoch 2/20\\n\",\n      \"9/9 - 1s - loss: 1.1236 - cls_label_loss: 1.1236 - cls_label_auc: 0.6073 - val_loss: 0.7250 - val_cls_label_loss: 0.7250 - val_cls_label_auc: 0.5605\\n\",\n      \"Epoch 3/20\\n\",\n      \"9/9 - 1s - loss: 0.7922 - cls_label_loss: 0.7922 - cls_label_auc: 0.5777 - val_loss: 0.7698 - val_cls_label_loss: 0.7698 - val_cls_label_auc: 0.5885\\n\",\n      \"Epoch 4/20\\n\",\n      \"9/9 - 1s - loss: 0.6654 - cls_label_loss: 0.6654 - cls_label_auc: 0.6627 - val_loss: 0.6365 - val_cls_label_loss: 0.6365 - val_cls_label_auc: 0.6055\\n\",\n      \"Epoch 5/20\\n\",\n      \"9/9 - 1s - loss: 0.5701 - cls_label_loss: 0.5701 - cls_label_auc: 0.7252 - val_loss: 0.6634 - val_cls_label_loss: 0.6634 - val_cls_label_auc: 0.6159\\n\",\n      \"Epoch 6/20\\n\",\n      \"9/9 - 1s - loss: 0.5389 - cls_label_loss: 0.5389 - cls_label_auc: 0.7541 - val_loss: 0.6247 - val_cls_label_loss: 0.6247 - val_cls_label_auc: 0.6402\\n\",\n      \"Epoch 7/20\\n\",\n      \"9/9 - 1s - loss: 0.5063 - cls_label_loss: 0.5063 - cls_label_auc: 0.7928 - val_loss: 0.6179 - val_cls_label_loss: 0.6179 - val_cls_label_auc: 0.6594\\n\",\n      \"Epoch 8/20\\n\",\n      \"9/9 - 1s - loss: 0.4940 - cls_label_loss: 0.4940 - cls_label_auc: 0.8012 - val_loss: 0.6137 - val_cls_label_loss: 0.6137 - val_cls_label_auc: 0.6710\\n\",\n      \"Epoch 9/20\\n\",\n      \"9/9 - 1s - loss: 0.4807 - cls_label_loss: 0.4807 - cls_label_auc: 0.8176 - val_loss: 0.6009 - val_cls_label_loss: 0.6009 - val_cls_label_auc: 0.6736\\n\",\n      \"Epoch 10/20\\n\",\n      \"9/9 - 1s - loss: 0.4478 - cls_label_loss: 0.4478 - cls_label_auc: 0.8687 - val_loss: 0.5882 - val_cls_label_loss: 0.5882 - val_cls_label_auc: 0.6880\\n\",\n      \"Epoch 11/20\\n\",\n      \"9/9 - 1s - loss: 0.4346 - cls_label_loss: 0.4346 - cls_label_auc: 0.8817 - val_loss: 0.5889 - val_cls_label_loss: 0.5889 - val_cls_label_auc: 0.6959\\n\",\n      \"Epoch 12/20\\n\",\n      \"9/9 - 1s - loss: 0.4192 - cls_label_loss: 0.4192 - cls_label_auc: 0.8963 - val_loss: 0.5838 - val_cls_label_loss: 0.5838 - val_cls_label_auc: 0.7004\\n\",\n      \"Epoch 13/20\\n\",\n      \"9/9 - 1s - loss: 0.4034 - cls_label_loss: 0.4034 - cls_label_auc: 0.9113 - val_loss: 0.5802 - val_cls_label_loss: 0.5802 - val_cls_label_auc: 0.7077\\n\",\n      \"Epoch 14/20\\n\",\n      \"9/9 - 1s - loss: 0.3892 - cls_label_loss: 0.3892 - cls_label_auc: 0.9186 - val_loss: 0.5747 - val_cls_label_loss: 0.5747 - val_cls_label_auc: 0.7162\\n\",\n      \"Epoch 15/20\\n\",\n      \"9/9 - 1s - loss: 0.3736 - cls_label_loss: 0.3736 - cls_label_auc: 0.9294 - val_loss: 0.5760 - val_cls_label_loss: 0.5760 - val_cls_label_auc: 0.7186\\n\",\n      \"Epoch 16/20\\n\",\n      \"9/9 - 1s - loss: 0.3601 - cls_label_loss: 0.3601 - cls_label_auc: 0.9389 - val_loss: 0.5731 - val_cls_label_loss: 0.5731 - val_cls_label_auc: 0.7215\\n\",\n      \"Epoch 17/20\\n\",\n      \"9/9 - 1s - loss: 0.3471 - cls_label_loss: 0.3471 - cls_label_auc: 0.9431 - val_loss: 0.5776 - val_cls_label_loss: 0.5776 - val_cls_label_auc: 0.7272\\n\",\n      \"Epoch 18/20\\n\",\n      \"9/9 - 1s - loss: 0.3372 - cls_label_loss: 0.3372 - cls_label_auc: 0.9480 - val_loss: 0.5731 - val_cls_label_loss: 0.5731 - val_cls_label_auc: 0.7282\\n\",\n      \"Epoch 19/20\\n\",\n      \"9/9 - 1s - loss: 0.3290 - cls_label_loss: 0.3290 - cls_label_auc: 0.9498 - val_loss: 0.5812 - val_cls_label_loss: 0.5812 - val_cls_label_auc: 0.7331\\n\",\n      \"Epoch 20/20\\n\",\n      \"9/9 - 1s - loss: 0.3190 - cls_label_loss: 0.3190 - cls_label_auc: 0.9516 - val_loss: 0.5695 - val_cls_label_loss: 0.5695 - val_cls_label_auc: 0.7372\\n\",\n      \"time: 23.8 s\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"finetune_history = model.fit(\\n\",\n    \"    train_generator,\\n\",\n    \"    epochs=EPOCHS,\\n\",\n    \"    validation_data=valid_generator,\\n\",\n    \"    verbose=2,\\n\",\n    \")\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"conventional-makeup\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"logical-opinion\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"median-beads\",\n   \"metadata\": {},\n   \"source\": [\n    \"# Cold Start\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 18,\n   \"id\": \"particular-corps\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"ename\": \"ValueError\",\n     \"evalue\": \"The first argument to `Layer.call` must always be passed.\",\n     \"output_type\": \"error\",\n     \"traceback\": [\n      \"\\u001b[0;31m---------------------------------------------------------------------------\\u001b[0m\",\n      \"\\u001b[0;31mValueError\\u001b[0m                                Traceback (most recent call last)\",\n      \"\\u001b[0;32m<ipython-input-18-e604b1f99c15>\\u001b[0m in \\u001b[0;36m<module>\\u001b[0;34m\\u001b[0m\\n\\u001b[1;32m      8\\u001b[0m     \\u001b[0mutil_vocab\\u001b[0m\\u001b[0;34m=\\u001b[0m\\u001b[0;36m4\\u001b[0m\\u001b[0;34m,\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m      9\\u001b[0m     \\u001b[0mdate_vocab\\u001b[0m\\u001b[0;34m=\\u001b[0m\\u001b[0;36m365\\u001b[0m\\u001b[0;34m,\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m---> 10\\u001b[0;31m     \\u001b[0mcat_vocab\\u001b[0m\\u001b[0;34m=\\u001b[0m\\u001b[0mlen\\u001b[0m\\u001b[0;34m(\\u001b[0m\\u001b[0mcat2int\\u001b[0m\\u001b[0;34m)\\u001b[0m\\u001b[0;34m,\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\\u001b[0m\\u001b[1;32m     11\\u001b[0m )\\n\",\n      \"\\u001b[0;32m/usr/local/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py\\u001b[0m in \\u001b[0;36m__call__\\u001b[0;34m(self, *args, **kwargs)\\u001b[0m\\n\\u001b[1;32m    940\\u001b[0m     \\u001b[0;31m#   not to any other argument.\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m    941\\u001b[0m     \\u001b[0;31m# - setting the SavedModel saving spec.\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m--> 942\\u001b[0;31m     \\u001b[0minputs\\u001b[0m\\u001b[0;34m,\\u001b[0m \\u001b[0margs\\u001b[0m\\u001b[0;34m,\\u001b[0m \\u001b[0mkwargs\\u001b[0m \\u001b[0;34m=\\u001b[0m \\u001b[0mself\\u001b[0m\\u001b[0;34m.\\u001b[0m\\u001b[0m_split_out_first_arg\\u001b[0m\\u001b[0;34m(\\u001b[0m\\u001b[0margs\\u001b[0m\\u001b[0;34m,\\u001b[0m \\u001b[0mkwargs\\u001b[0m\\u001b[0;34m)\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\\u001b[0m\\u001b[1;32m    943\\u001b[0m     \\u001b[0minput_list\\u001b[0m \\u001b[0;34m=\\u001b[0m \\u001b[0mnest\\u001b[0m\\u001b[0;34m.\\u001b[0m\\u001b[0mflatten\\u001b[0m\\u001b[0;34m(\\u001b[0m\\u001b[0minputs\\u001b[0m\\u001b[0;34m)\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m    944\\u001b[0m \\u001b[0;34m\\u001b[0m\\u001b[0m\\n\",\n      \"\\u001b[0;32m/usr/local/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py\\u001b[0m in \\u001b[0;36m_split_out_first_arg\\u001b[0;34m(self, args, kwargs)\\u001b[0m\\n\\u001b[1;32m   3046\\u001b[0m     \\u001b[0;32melse\\u001b[0m\\u001b[0;34m:\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m   3047\\u001b[0m       raise ValueError(\\n\\u001b[0;32m-> 3048\\u001b[0;31m           'The first argument to `Layer.call` must always be passed.')\\n\\u001b[0m\\u001b[1;32m   3049\\u001b[0m     \\u001b[0;32mreturn\\u001b[0m \\u001b[0minputs\\u001b[0m\\u001b[0;34m,\\u001b[0m \\u001b[0margs\\u001b[0m\\u001b[0;34m,\\u001b[0m \\u001b[0mkwargs\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m   3050\\u001b[0m \\u001b[0;34m\\u001b[0m\\u001b[0m\\n\",\n      \"\\u001b[0;31mValueError\\u001b[0m: The first argument to `Layer.call` must always be passed.\"\n     ]\n    },\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 58.9 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"m = model(\\n\",\n    \"    patient_dim=PATIENT_DIM,\\n\",\n    \"    max_visit=MAX_VISIT,\\n\",\n    \"    max_code=MAX_CODE,\\n\",\n    \"    max_demo=MAX_DEMO,\\n\",\n    \"    code_vocab=len(code2int),\\n\",\n    \"    demo_vocab=len(demo2int),\\n\",\n    \"    util_vocab=4,\\n\",\n    \"    date_vocab=365,\\n\",\n    \"    cat_vocab=len(cat2int),\\n\",\n    \")\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"demonstrated-congress\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 19,\n   \"id\": \"frozen-borough\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"ename\": \"NameError\",\n     \"evalue\": \"name 'm' is not defined\",\n     \"output_type\": \"error\",\n     \"traceback\": [\n      \"\\u001b[0;31m---------------------------------------------------------------------------\\u001b[0m\",\n      \"\\u001b[0;31mNameError\\u001b[0m                                 Traceback (most recent call last)\",\n      \"\\u001b[0;32m<ipython-input-19-440ef62b5236>\\u001b[0m in \\u001b[0;36m<module>\\u001b[0;34m\\u001b[0m\\n\\u001b[0;32m----> 1\\u001b[0;31m m.compile(optimizer=\\\"RMSprop\\\", loss=tf.keras.losses.BinaryCrossentropy(), metrics=[tf.keras.metrics.Recall(top_k=5), \\n\\u001b[0m\\u001b[1;32m      2\\u001b[0m                                                                                        \\u001b[0mtf\\u001b[0m\\u001b[0;34m.\\u001b[0m\\u001b[0mkeras\\u001b[0m\\u001b[0;34m.\\u001b[0m\\u001b[0mmetrics\\u001b[0m\\u001b[0;34m.\\u001b[0m\\u001b[0mRecall\\u001b[0m\\u001b[0;34m(\\u001b[0m\\u001b[0mtop_k\\u001b[0m\\u001b[0;34m=\\u001b[0m\\u001b[0;36m10\\u001b[0m\\u001b[0;34m)\\u001b[0m\\u001b[0;34m,\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m      3\\u001b[0m                                                                                        tf.keras.metrics.Recall(top_k=30)])\\n\\u001b[1;32m      4\\u001b[0m \\u001b[0mprint\\u001b[0m\\u001b[0;34m(\\u001b[0m\\u001b[0mm\\u001b[0m\\u001b[0;34m.\\u001b[0m\\u001b[0msummary\\u001b[0m\\u001b[0;34m(\\u001b[0m\\u001b[0;34m)\\u001b[0m\\u001b[0;34m)\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\",\n      \"\\u001b[0;31mNameError\\u001b[0m: name 'm' is not defined\"\n     ]\n    },\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 16.8 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"m.compile(optimizer=\\\"RMSprop\\\", loss=tf.keras.losses.BinaryCrossentropy(), metrics=[tf.keras.metrics.Recall(top_k=5), \\n\",\n    \"                                                                                       tf.keras.metrics.Recall(top_k=10), \\n\",\n    \"                                                                                       tf.keras.metrics.Recall(top_k=30)])\\n\",\n    \"print(m.summary())\\n\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 26,\n   \"id\": \"structural-space\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"ename\": \"NameError\",\n     \"evalue\": \"name 'm' is not defined\",\n     \"output_type\": \"error\",\n     \"traceback\": [\n      \"\\u001b[0;31m---------------------------------------------------------------------------\\u001b[0m\",\n      \"\\u001b[0;31mNameError\\u001b[0m                                 Traceback (most recent call last)\",\n      \"\\u001b[0;32m<ipython-input-26-ea420610e986>\\u001b[0m in \\u001b[0;36m<module>\\u001b[0;34m\\u001b[0m\\n\\u001b[1;32m      1\\u001b[0m \\u001b[0mopt\\u001b[0m \\u001b[0;34m=\\u001b[0m \\u001b[0mtf\\u001b[0m\\u001b[0;34m.\\u001b[0m\\u001b[0mkeras\\u001b[0m\\u001b[0;34m.\\u001b[0m\\u001b[0moptimizers\\u001b[0m\\u001b[0;34m.\\u001b[0m\\u001b[0mAdam\\u001b[0m\\u001b[0;34m(\\u001b[0m\\u001b[0mlearning_rate\\u001b[0m\\u001b[0;34m=\\u001b[0m\\u001b[0;36m0.01\\u001b[0m\\u001b[0;34m)\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m----> 2\\u001b[0;31m m.compile(optimizer=opt, loss=tf.keras.losses.BinaryCrossentropy(), metrics=[tf.keras.metrics.Recall(top_k=5), \\n\\u001b[0m\\u001b[1;32m      3\\u001b[0m                                                                                        \\u001b[0mtf\\u001b[0m\\u001b[0;34m.\\u001b[0m\\u001b[0mkeras\\u001b[0m\\u001b[0;34m.\\u001b[0m\\u001b[0mmetrics\\u001b[0m\\u001b[0;34m.\\u001b[0m\\u001b[0mRecall\\u001b[0m\\u001b[0;34m(\\u001b[0m\\u001b[0mtop_k\\u001b[0m\\u001b[0;34m=\\u001b[0m\\u001b[0;36m10\\u001b[0m\\u001b[0;34m)\\u001b[0m\\u001b[0;34m,\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m      4\\u001b[0m                                                                                        tf.keras.metrics.Recall(top_k=30)])\\n\\u001b[1;32m      5\\u001b[0m \\u001b[0mprint\\u001b[0m\\u001b[0;34m(\\u001b[0m\\u001b[0mm\\u001b[0m\\u001b[0;34m.\\u001b[0m\\u001b[0msummary\\u001b[0m\\u001b[0;34m(\\u001b[0m\\u001b[0;34m)\\u001b[0m\\u001b[0;34m)\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\",\n      \"\\u001b[0;31mNameError\\u001b[0m: name 'm' is not defined\"\n     ]\n    },\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 18.9 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"opt = tf.keras.optimizers.Adam(learning_rate=0.01)\\n\",\n    \"m.compile(optimizer=opt, loss=tf.keras.losses.BinaryCrossentropy(), metrics=[tf.keras.metrics.Recall(top_k=5), \\n\",\n    \"                                                                                       tf.keras.metrics.Recall(top_k=10), \\n\",\n    \"                                                                                       tf.keras.metrics.Recall(top_k=30)])\\n\",\n    \"print(m.summary())\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"cleared-swimming\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"cold_start_his = m.fit(\\n\",\n    \"    train_generator,\\n\",\n    \"    epochs=EPOCHS,\\n\",\n    \"    validation_data=valid_generator,\\n\",\n    \"    verbose=2,\\n\",\n    \")\\n\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"apparent-percentage\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"shaped-variable\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"japanese-canvas\",\n   \"metadata\": {},\n   \"source\": [\n    \"# plot\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 21,\n   \"id\": \"comprehensive-playback\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 218 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"import matplotlib.pyplot as plt\\n\",\n    \"\\n\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 22,\n   \"id\": \"established-probe\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"ename\": \"KeyError\",\n     \"evalue\": \"'val_code_label_recall_19'\",\n     \"output_type\": \"error\",\n     \"traceback\": [\n      \"\\u001b[0;31m---------------------------------------------------------------------------\\u001b[0m\",\n      \"\\u001b[0;31mKeyError\\u001b[0m                                  Traceback (most recent call last)\",\n      \"\\u001b[0;32m<ipython-input-22-d990d1626ccd>\\u001b[0m in \\u001b[0;36m<module>\\u001b[0;34m\\u001b[0m\\n\\u001b[0;32m----> 1\\u001b[0;31m \\u001b[0mft_recall_5\\u001b[0m \\u001b[0;34m=\\u001b[0m \\u001b[0;34m[\\u001b[0m\\u001b[0;36m0.2107\\u001b[0m\\u001b[0;34m]\\u001b[0m \\u001b[0;34m+\\u001b[0m \\u001b[0mfinetune_history\\u001b[0m\\u001b[0;34m.\\u001b[0m\\u001b[0mhistory\\u001b[0m\\u001b[0;34m[\\u001b[0m\\u001b[0;34m'val_code_label_recall_19'\\u001b[0m\\u001b[0;34m]\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\\u001b[0m\\u001b[1;32m      2\\u001b[0m \\u001b[0mft_recall_10\\u001b[0m \\u001b[0;34m=\\u001b[0m \\u001b[0;34m[\\u001b[0m\\u001b[0;36m0.3366\\u001b[0m\\u001b[0;34m]\\u001b[0m \\u001b[0;34m+\\u001b[0m \\u001b[0mfinetune_history\\u001b[0m\\u001b[0;34m.\\u001b[0m\\u001b[0mhistory\\u001b[0m\\u001b[0;34m[\\u001b[0m\\u001b[0;34m'val_code_label_recall_20'\\u001b[0m\\u001b[0;34m]\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m      3\\u001b[0m \\u001b[0mft_recall_30\\u001b[0m \\u001b[0;34m=\\u001b[0m \\u001b[0;34m[\\u001b[0m\\u001b[0;36m0.5424\\u001b[0m\\u001b[0;34m]\\u001b[0m \\u001b[0;34m+\\u001b[0m \\u001b[0mfinetune_history\\u001b[0m\\u001b[0;34m.\\u001b[0m\\u001b[0mhistory\\u001b[0m\\u001b[0;34m[\\u001b[0m\\u001b[0;34m'val_code_label_recall_21'\\u001b[0m\\u001b[0;34m]\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m      4\\u001b[0m \\u001b[0;34m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m      5\\u001b[0m \\u001b[0mcs_recall_5\\u001b[0m \\u001b[0;34m=\\u001b[0m \\u001b[0mcold_start_his\\u001b[0m\\u001b[0;34m.\\u001b[0m\\u001b[0mhistory\\u001b[0m\\u001b[0;34m[\\u001b[0m\\u001b[0;34m'val_recall_41'\\u001b[0m\\u001b[0;34m]\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\",\n      \"\\u001b[0;31mKeyError\\u001b[0m: 'val_code_label_recall_19'\"\n     ]\n    },\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 19.9 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"ft_recall_5 = [0.2107] + finetune_history.history['val_code_label_recall_19']\\n\",\n    \"ft_recall_10 = [0.3366] + finetune_history.history['val_code_label_recall_20']\\n\",\n    \"ft_recall_30 = [0.5424] + finetune_history.history['val_code_label_recall_21']\\n\",\n    \"\\n\",\n    \"cs_recall_5 = cold_start_his.history['val_recall_41']\\n\",\n    \"cs_recall_10 = cold_start_his.history['val_recall_42']\\n\",\n    \"cs_recall_30 = cold_start_his.history['val_recall_43']\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"ordered-fraud\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"value1 = ft_recall_5\\n\",\n    \"value2 = cs_recall_5\\n\",\n    \"\\n\",\n    \"length = len(value1)\\n\",\n    \"plt.plot(range(length), value1, \\\"-s\\\", label=\\\"Finetune\\\")\\n\",\n    \"plt.plot(range(1, length), value2, \\\"-s\\\", label=\\\"w/o Pretrain\\\")\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"plt.xticks(range(length))\\n\",\n    \"plt.ylabel('Recall@5')\\n\",\n    \"plt.xlabel(\\\"EPOCHS\\\")\\n\",\n    \"plt.legend()\\n\",\n    \"plt.show()\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 151,\n   \"id\": \"gothic-evaluation\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"image/png\": \"iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA8Z0lEQVR4nO3deXxU1fn48c+TnX0N+yoCsoggERRFkEVxC1qxoFhxaa0Lrq1Wf1pLrbZWtN9qtSpuuFBRcUMFARXrAihBQVkkLGUJSwh7Qsj+/P64NzCGmcnMJJOZSZ736zWvzNx7z73PJJP7zLnn3HNEVTHGGGMqiot0AMYYY6KTJQhjjDFeWYIwxhjjlSUIY4wxXlmCMMYY41VCpAOoLi1bttQuXbpEOgxjjIkpy5Yt262qqd7W1ZoE0aVLFzIyMiIdhjHGxBQR2exrnV1iMsYY45UlCGOMMV5ZgjDGGOOVJQhjjDFehTVBiMgYEVkrIutF5G4v668XkR9FZLmIfCUivd3lo0VkmbtumYiMCGecxhhjjhW2XkwiEg88BYwGsoClIjJbVVd7bPYfVX3G3T4d+AcwBtgNXKiq20WkLzAPaB+uWI0xJhLSHlzA7ryiY5a3bJhExn2jIxDRz4WzBjEIWK+qG1W1CJgJjPXcQFUPerxsAKi7/HtV3e4uXwXUE5HkMMZqjDE1zlty8Le8poXzPoj2wFaP11nA4IobichNwB1AEuDtUtIlwHeqWhiOII0xsS+S38SDPbaqcrCghOyDBX73uz+/iCb1EhGRajluKCJ+o5yqPgU8JSKXA/cBk8rXiUgf4O/A2d7Kish1wHUAnTp1Cn+wxpiwqcoJryrfxKt6ovV37Be/+h/ZBwvYebCAnQcKyD5YQPbBQg4Xl1a63/4PLCAxXkhtmExqI89HCqmNkmuk9hHOBLEN6OjxuoO7zJeZwNPlL0SkA/AucKWqbvBWQFWnAdMA0tLSbOYjYyKspk7y+UUl7MkrYs+hIvbk+b+48MriTdRLjKd+UgL1k+JJSYynfpLzqJcU7/e4323Zx8HDxRwsKHF/FnPwcAkHjjwv9nvsBz5cTVJ8HK2bJNOmcQp92zdhZK8U2jROoXWTFG55/XufZf94QW9ycgudR14h2/YXsHzrfvYcKqKm5nkLZ4JYCnQXka44iWECcLnnBiLSXVXXuS/PB9a5y5sCHwF3q+rXYYzRGFNBuE7yqkpRaRmFJWUUFpdRWFL6s+f+XPXSt+w9VOQmhUIKissCfj/3v78q4G0r+sW/Fx2zLCk+jsb1EmlcL4HGKYl+y3/3x9E0q+/7MpG/BHHtGV29Li8pLWPvoSIG/fVTv8euDmFLEKpaIiKTcXogxQMvquoqEXkAyFDV2cBkERkFFAP7OHp5aTJwPHC/iNzvLjtbVXeFK15jjMPfSX7F1v1HvkWXf4P2fO1P13vmVCGmQlo0SOb41Ia0aJhE8wbJtGiQ5D5P4mIvJ/JyGfeN4nBRKflFpeQXlXC4uPTI68NFpdz19g8+y06/+hQnGaQcTQgpifE/26bL3R/5LN+8QZLf99WyYZLPZOxLQnwcrRqn+N1vdQlrG4SqzgHmVFh2v8fzW32UexB4MJyxGVNbhdJouuNAARty8li/K8/vvsc+dWyFPj5OaJySQON6/r9N3zLieJIT40lOiDv6MyGO5IR4khPjuPqlpT7LfnjzUL/79qdlQ/8dIP0liOE9W4V83EBEQ1dWfyLeSG2MqV7+agBrd+ayISePDbvyWJ+Tx4acPDbmHCK/qPJGU4Dnr0yjSf2ff6OunxR/5BKKv2/Td5zdM/g3E6BQvonH+rFr4riWIIyJQqG0A5SUlpFTSYPtOf/84sjz9k3r0a1VQ07p0pxuqQ2dR6sGDHrI97XtUb1bB/gOgleVE15VvolX9UQbqVpATRzXEoQxYRKuxt43M7aSfaCA7NwCdh4oPNKNcndeYaW9Wx6f0J9uqQ05LrUB9ZOq/98/Uif5qoj2yzyRZAnCmDCprNtmcWkZe/KK3G6MBUe7NOb6rwXcNcu5Zt60fqLTXbJxCr3bNqZ142RaN0nh3ndX+iw7tn/lI9bE4knehIclCGMiYOBfFrA333t/9iaVNPZ+cedZtGqcfExvmnL+EkQg7CRvylmCMKaalJSWsWr7Qb79316+3bTX77Zj+rY5emesx52yLRs6J35/jb2dWtT3u+9INtia2sUShDF++GtH+OoPI1i+dT9L3YTw3eZ9HHJ7A3Wp5CT+0MUnhiVesBqAqT6WIIzxw187Qr8p8ykqLUMEerZuxCUDOzCoa3MGdWlOq8YpfmsBlbFagIkGliBMrRZsT6JDhSVk7TtM1r58svYd9rvvq0/vwildmpPWpRlN6x974rbGXhPrLEGYWs1fDeCVxZt+lgy27s1nX77/4SI83XNeL7/r7SRvYp0lCBP1Aq0FlJYpu3IL2L6/gO37D7PjgP8awP3vryIpIY4OzerRoVl9+rZvQodm9ejYrP6RZac89Em1vx9jYoUlCBP1/NUCbnn9ezcZODeLlZYFPg7yt/eOpGWDZOLivI+0aUxdZwnCRC1VZV0lg8ct37qftk1SGNy1OW2bptCuaT3aNalHu6b1aNs0hX5T5vss26pR5SNiWmOxqcssQZiosjuvkK/X7+aLzN18tT6H7IP+7yr+4q6zwhqPtSOYuswShKkR/toR/jl+AF+uz+HLzN2s3nEQcIaROOP4lgzt3pI/vP1jyMe1GoAxobMEYWqEv3aEK174hsR4YWDnZtx5Tk+Gdm9Jn3ZNiHfbBqqSIKwGYEzoLEGYsKus4filq05hUNfmNEj2/nG0WoAxkWEJwoRFcWkZSzbuYe7KncxftdPvtmed4H/WLqsFGBMZliBMtSksKeWrdbuZu3Inn6zJZn9+MfWT4jmrZys++nFHpMMzxgTJEoQJmK+G5sYpCZx1Qis+W7OL3MISGqUkMKpXa8b0bcOwHqmkJMbzURXGJTLGREZYE4SIjAEeB+KB51X14QrrrwduAkqBPOA6VV3trrsHuNZdd4uqzgtnrKZyvhqaDxaU8EVmDued2JYxJ7bh9G4tSUqI+9k21o5gTOwJW4IQkXjgKWA0kAUsFZHZ5QnA9R9VfcbdPh34BzBGRHoDE4A+QDvgExHpoaqBzaxuqp1WMpfl0ntHkRAf53O9tSMYE3vCWYMYBKxX1Y0AIjITGAscSRCqetBj+wZA+VloLDBTVQuB/4nIend/i8MYr/Fif34R736/jde/3eJ3O3/JwRgTm8KZINoDWz1eZwGDK24kIjcBdwBJwAiPsksqlD1mMl0RuQ64DqBTp07VErRxagtLN+3j9W+38NGPOygqKeOkjk0jHZYxpoZFvJFaVZ8CnhKRy4H7gElBlJ0GTANIS0sLfJQ249XeQ0W8810Wr3+7hQ05h2iUnMD4tI5MGNSRPu2aVGkCHGNM7AlngtgGdPR43cFd5stM4OkQy5oA+eqJlJwQhyoUlZYxoFNTHhnXjwv6taV+0tGPiDU0G1O3hDNBLAW6i0hXnJP7BOByzw1EpLuqrnNfng+UP58N/EdE/oHTSN0d+DaMsdYZvnoiFZaUcdWQLkwY1JET2jT2uo01NBtTt4QtQahqiYhMBubhdHN9UVVXicgDQIaqzgYmi8gooBjYh3t5yd3uTZwG7RLgJuvBVHWVDXkxJb1PDUVijIkFYW2DUNU5wJwKy+73eH6rn7IPAQ+FL7q6Y+vefN7K2MqsZVmRDsUYE0Mi3khtwqOguJT5q7N5c+lWvt6wG4Ch3VPZfqAgwpEZY2KFJYhaZvX2g7yZsZV3v9/GgcPFtG9aj9tG9mBcWgfaN61nPZGMMQGzBBFjfPVCapgcz3GpDfkh6wBJ8XGc3ac1E07pxJBuLX4257L1RDLGBMoSRIzx1Qspr7CUopIy/nRhby7q355mDbyf8K0nkjEmUJYgapG5tw5FRCrf0BhjAmAJIkZs3ZvPK4s3+d3GkoMxpjpZgohiqsqiDXuYvmgTn6zJJt4SgDGmBlmCiEL5RSW8+/02Xl60iczsPFo0SGLyWcczcXBnTv3bp5EOzxhTR1iCiABfPZGa109kXFpHZn67hYMFJfRp15hHLz2JC/q1JSUxHrBeSMaYmmMJIgJ89UTam1/MC1/9jzF923D1kC4M7NzsmHYF64VkjKkpliCizFd/OIu2TepFOgxjjMGmAYsylhyMMdHCEoQxxhivLEHUsKKSskiHYIwxAbEEUcOmfLDK5zrriWSMiSbWSF2DXl2ymf98s4Xrh3Xj7nNPiHQ4xhjjl9UgasiSjXv48+xVjDihFXee0zPS4RhjTKUsQdSArXvzuXHGd3RuUZ9/TuhPfJwNmWGMiX6WIMLsUGEJv3klg+LSMp67Mo3GKYmRDskYYwIS1gQhImNEZK2IrBeRu72sv0NEVovIDyLyqYh09lj3iIisEpE1IvKExOBQpWVlyu/fWkFmdi5PXn4yx6U2jHRIxhgTsLAlCBGJB54CzgV6A5eJSO8Km30PpKlqP2AW8IhbdghwOtAP6AucAgwLV6zh8q/P1jN35U7uObcXw3qkRjocY4wJSjhrEIOA9aq6UVWLgJnAWM8NVHWhqua7L5cAHcpXASlAEpAMJALZYYy12n28cif/90kmvxjQnl8P7RrpcIwxJmjhTBDtga0er7PcZb5cC8wFUNXFwEJgh/uYp6prKhYQketEJENEMnJycqot8Kr6aedB7nhzOSd1bMpff3GiTeRjjIlJUdFILSJXAGnAVPf18UAvnBpFe2CEiAytWE5Vp6lqmqqmpaZGxyWcvYeK+M0rGTRMTmDarwYeGabbGGNiTTgTxDago8frDu6ynxGRUcC9QLqqFrqLLwaWqGqequbh1CxOC2Os1aK4tIybZnxH9sFCnv3VQFo3Tol0SMYYE7JwJoilQHcR6SoiScAEYLbnBiIyAHgWJzns8li1BRgmIgkikojTQH3MJaZo85cPV7N44x7+dvGJDOjULNLhGGNMlYQtQahqCTAZmIdzcn9TVVeJyAMiku5uNhVoCLwlIstFpDyBzAI2AD8CK4AVqvpBuGKtDq9/u4VXFm/m12d05ZKBHSovYIwxUU5UNdIxVIu0tDTNyMioueP5mDa0ZcMkm/XNGBMzRGSZqqZ5WxcVjdSxyNe0ob6WG2NMrLEEYYwxxitLEMYYY7yyBGGMMcYrSxDGGGO8sgQRIl/Tg9q0ocaY2sKmHA1Rxn2juf7VZWRm5/LZ74dHOhxjjKl2VoOogsxduXRvbXM8GGNqJ0sQISooLmXT7kP0bN0o0qEYY0xYWIII0cacQ5QpdLcEYYyppSxBhCgzOxeAnm0sQRhjaidLECHKzM4lIU7o0qJBpEMxxpiwsAQRoszsPLq2bEBSgv0KjTG1k53dQpSZnUsPu7xkjKnFLEGEIL+ohK378unRyhKEMab2sgQRgvW78lCFnm3sHghjTO1lCSIEmdl5gHVxNcbUbkEnCHeO6V+IyAnhCCgWZGbnkpQQR+fm9SMdijHGhE2lCUJE3vN4Phb4DLgQeF9ErgpbZFEsMzuXbqkNSYi3CpgxpvYK5AzX2eP5H4ARqno1cDpwu7+CIjJGRNaKyHoRudvL+jtEZLWI/CAin4pIZ491nURkvoiscbfpEthbCr/Mnbn0tDGYjDG1XCAJQj2eJ6jq/wBUdTdQ5quQiMQDTwHnAr2By0Skd4XNvgfSVLUfMAt4xGPdK8BUVe0FDAJ2BRBr2OUWFLP9QIG1Pxhjar1Ahvs+SUQOAgIki0hbVd0hIklAvJ9yg4D1qroRQERmAmOB1eUbqOpCj+2XAFe42/bGSUYL3O3ygnhPYVXeQG2D9BkTvOLiYrKysigoKIh0KHVOSkoKHTp0IDExMeAylSYIVfWVBOoDv/VTtD2w1eN1FjDYz/bXAnPd5z2A/SLyDtAV+AS4W1VLPQuIyHXAdQCdOnXys+vqs84dg6mHJQhjgpaVlUWjRo3o0qULIhLpcOoMVWXPnj1kZWXRtWvXgMuF0ouphYjEqep+VV0cbHkf+7wCSAOmuosSgKHA74FTgOOAqyqWU9VpqpqmqmmpqanVEUql1mbnUi8xng7N6tXI8YypTQoKCmjRooUlhxomIrRo0SLomltAM8qJSDPgL8CJwA6gmYhsA25W1UM+im0DOnq87uAuq7jvUcC9wDBVLXQXZwHLPS5PvQecCrwQSLzhtC47j+6tGxIXZx/woEztDoe8NCM1aAV3rqv5eEzEWHKIjFB+74F0c20KzAHeVtVhqjpBVc8BXgUeFpGhItLcS9GlQHf3vokkYAIwu8K+BwDPAumquqtC2aYiUl4tGIFH20Ukrc3OtctLofCWHPwtNyZM4uPj6d+//5HHpk2bGDJkSMj7mz59Otu3b6/GCKNHIDWIPwKPqupCEXkV55v8bqAl8CNO4/V9wB2ehVS1REQmA/NwGrNfVNVVIvIAkKGqs3EuKTUE3nKz2xZVTVfVUhH5PfCpOCuWAc9Vw/utkn2HisjJLaSHdXE1JuzSHlzA7ryiY5a3bJhExn2jQ95vvXr1WL58+c+WLVq0KOT9TZ8+nb59+9KuXbuQ9xGtAmmDOFNV33afFwKXqeppwHhgD/AVcJa3gqo6R1V7qGo3VX3IXXa/mxxQ1VGq2lpV+7uPdI+yC1S1n6qeqKpXqeqxn5QalmkN1OGhWvk2ps7xlhz8La+Khg2dL32ff/45w4cPZ9y4cZxwwglMnDgRdT+fy5YtY9iwYQwcOJBzzjmHHTt2MGvWLDIyMpg4cSL9+/fn8OHDdOnShd27dwOQkZHB8OHDAZgyZQrXXHMNw4cP57jjjuOJJ544cvzXXnuNQYMG0b9/f377299SWlpKNAikBpEiIqLOb+lkYIW7fCVwsqqW1ZVripm7nC6uliCCVFkC+Gc/6HUh9E6HDoMgzu5Qrwv+/MEqVm8/GFLZ8c967x/Tu11j/nRhH79lDx8+TP/+/QHo2rUr77777s/Wf//996xatYp27dpx+umn8/XXXzN48GBuvvlm3n//fVJTU3njjTe49957efHFF3nyySd59NFHSUtLqzTun376iYULF5Kbm0vPnj254YYbWL9+PW+88QZff/01iYmJ3HjjjcyYMYMrr7wysF9GGAWSIL4FRuJ0Nf03MF9EFgOnAc+KyCnAqvCFGD3WZefSKDmBtk1SIh1K7CgphA9u9b9N696w9DlY8hQ0bAO9LoBe6dD5dIhPsAZuU628XWLyNGjQIDp06ABwpI2iadOmrFy5ktGjnUtbpaWltG3bNuhjn3/++SQnJ5OcnEyrVq3Izs7m008/ZdmyZZxyyimAk8BatWoV/BsLg0ASxEPAmyJyvqo+7/YoOg74B84lqtnApPCFGD3W7syle+uG1gsjUId2w8yJsHUJJDaAYi8d3hq0gsvfgIKDsG4+rH4fvp8BS5+H+i2g53nWwF1LVfZNv8vdH/lc98ZvT6vucI5ITk4+8jw+Pp6SkhJUlT59+rB4ceU9+xMSEigrcwaZqNit1Ne+J02axN/+9rdqegfVJ5Ab5TaKyE3AbBGZj3PHcylwnvv4naquDW+YkaeqZGbnMqZvm0iHEhuyV8Pr4yFvF4x7Efpe4n/7lMZw4jjnUZQP6z+BNbNh1Xs1Eq4x/vTs2ZOcnBwWL17MaaedRnFxMZmZmfTp04dGjRqRm5t7ZNsuXbqwbNkyzj33XN5++20/e3WMHDmSsWPHcvvtt9OqVSv27t1Lbm4unTt3rrRsuAV0sVdVv8G5pPQF0Avoi5Mohqjql+ELL3rszitiX34x3W0WucplzocXzoaSIrh6TuXJoaKk+k57xCXPw10bwhOjiXotGyYFtTyckpKSmDVrFn/4wx846aST6N+//5GeT1dddRXXX3/9kUbqP/3pT9x6662kpaURH+9vNCJH7969efDBBzn77LPp168fo0ePZseOHeF+SwERrSU9SNLS0jQjIyNs+1+0fjeXP/8NM349mNOPbxm248Q0VVjyb5h/H7TuC5fNhCbtq77fKU38rDtQ9f2bGrNmzRp69eoV6TDqLG+/fxFZpqpeW9grvcQkIrk4I7oKPx/ZVQBV1cahhxs71rpdXLvbPRDelRTBnN/Ddy87PZIufhaSGoT/uGVl1uvJmDCp9D9LVRupamOPn409X9dEkNEgMzuPpvUTSW2YXPnGdU3+XnjtF05yGPo7uPSV6k0ODfz06Hj/Rigtqb5jGWOOCKQG4W0YjSNUdW/1hRO9Mt0hNqwHUwU5mfCfX8LBbXDxNDhpfPUfw1tXVlX44lFY+CAUHIBxL0GidT82pjoF0s11GUcvMVWkOF1ea7XyHkxj+9e+W+mD4ut+BASunQ8dB9VcLCIw7E5IaQJz74QZ4+Cy1yHZOhEYU10C6eYa+ODhtdTOgwXkFpTYJEE+7zvQmk0OngZfB/WawrvXw8vpcMXbUN9vpdcYE6CAhvsu5w773R04UpdX1S+qO6hoUz6LnE0zGqX6/dKpObx1Fbx0LvzqXWhcx2t7xlSDgLt/iMivce6DmAf82f05JTxhRZfMnXV8kD5V2LCw8u0iqee5Tu3hwDZ44RzYY/dPmNA9/PDDzJgxI6Btp0+fTmpqKv3796d3794891xwA0+/9957rF4d/GwGs2fP5uGHHw66XDCCqUHcijO72xJVPUtETgD+Gp6woktmdi4tGybTvEHN36ATUQUHYPnrzrAXe2JgzKMuZ8Ck2fDaJfDiGKcm0aZvpKMyoYrgGFzz5s3jzTffDHj78ePH8+STT7Jr1y769OlDeno6rVu3PrK+pKSEhATvp9v33nuPCy64gN69ex+zzl+59PR00tPTva6rLsF0IC9Q1QIAEUlW1Z+AnuEJK7pkZufSs00duv8hexV8cBs81gs+/oPTEHzxs5GOKjDtT4ZrPob4RJh+Hmz5JtIRmVCFYQyuqVOnHhlm+/bbb2fEiBEAfPbZZ0ycOBGAgwcPUlRURGpqKps2bWLEiBH069ePkSNHsmXLFr/7b9WqFd26dWPz5s1H7rAePHgwd911Fxs2bGDMmDEMHDiQoUOH8tNPP7Fo0SJmz57NnXfeSf/+/dmwYQPDhw/ntttuIy0tjccff5wPPviAwYMHM2DAAEaNGkV2djbg1FwmT54MOHdz33LLLQwZMoTjjjuOWbNmhfw78hRMDSLLnV3uPWCBiOwDNldLFFGsrExZtyuPX6Z1rHzjWFZaDGs+cGoLm7+GhBToOw4G/RraDXC2mf9H39/ooklqTydJvHIRvHoRjH8Vjh8V6ahMRXPvhp0/hlb2pfO9L29zIpzr+7LL0KFDeeyxx7jlllvIyMigsLCQ4uJivvzyS84880wAPvnkE0aOHAnAzTffzKRJk5g0aRIvvvgit9xyC++9957P/W/cuJGNGzdy/PHHA5CVlcWiRYuIj49n5MiRPPPMM3Tv3p1vvvmGG2+8kc8++4z09HQuuOACxo0bd2Q/RUVFlI8MsW/fPpYsWYKI8Pzzz/PII4/w2GOPHXPsHTt28NVXX/HTTz+Rnp7+s/2FKuAEoaoXu0+niMhCoAnwcZUjiHLb9h8mv6iUnm1qSfuDz66qcUAZNO0Mox+AAb86tjdQLA2t3bSTkyT+0cu55FSRDRVeJw0cOJBly5Zx8OBBkpOTOfnkk8nIyODLL788UrP4+OOPufrqqwFYvHgx77zzDgC/+tWvuOuuu7zu94033uCrr74iOTmZZ599lubNnf+dSy+9lPj4ePLy8li0aBGXXnrpkTKFhYU+4xw//uj9RFlZWYwfP54dO3ZQVFRE167eO5ZedNFFxMXF0bt37yO1jKoKOEGIyKnAKlXNVdX/ikhjYABQq+vwR2eRqyWXmHxWz8vg8jedb9pxlQ8wFhMatoIyH3dZ21Dhkefnmz7gfwyuq30PBe5PYmIiXbt2Zfr06QwZMoR+/fqxcOFC1q9ff2SMom+//Zann346qP2Wt0FU1KCBM6JAWVkZTZs29TsPhbdy4NRi7rjjDtLT0/n888+ZMmWK1zKeQ4lX1xh7wbRBPA3kebzOc5fVanWqi2uPc2pPcjDGh6FDh/Loo49y5plnMnToUJ555hkGDBiAiLBq1SpOOOGEI6OwDhkyhJkzZwIwY8YMhg4dGtIxGzduTNeuXXnrrbcA5wS+YoUzOWfF4cIrOnDgAO3bO4NevvzyyyEdP1TBJIjyaUcBUNUyKqmBiMgYEVkrIutF5G4v6+8QkdUi8oOIfCoinSusbywiWSJybGquIZnZubRtkkLjlMRIhWDC5cC2SEdg/PHVtlXFNq+hQ4eyY8cOTjvtNFq3bk1KSsqRE//cuXMZM2bMkW3/9a9/8dJLL9GvXz9effVVHn/88ZCPO2PGDF544QVOOukk+vTpw/vvvw/AhAkTmDp1KgMGDGDDhmO7Z0+ZMoVLL72UgQMH0rJlzY4kHfBw3yLyDvA5R2sNNwJnqepFPraPBzKB0UAWsBS4TFVXe2xzFvCNquaLyA3AcFUd77H+cSAV2Kuqk/3FF67hvs9/4ktaNkzm5WsidKdwdatrQ2f7e78JKTD4ejjjdudubBN20T7c9+jRo3nllVdCmk40FgQ73HcwNYjrgSHANpwT/mDgOj/bDwLWq+pGVS0CZgJjPTdQ1YWqmu++XAJ08Ah6INAamB9EjNWqtExZvyuv9rQ/rJ0b6QiiS++x8PXj8ER/WPyUM3+2qdMWLFhQa5NDKAJOEKq6S1UnqGorVW2tqperqr+WvvbAVo/XWe4yX64F5gKISBzwGPB7fzGJyHUikiEiGTk5OYG9kSBs2ZtPYUlZ7biD+vB+594G8dHGEG1dVauLv8sUv5gGv/3C6cY77//Bk2nww1vOHBPGmKB6MfXAubzUWlX7ikg/IF1VH6xqECJyBZAGDHMX3QjMUdUsf8Nrq+o0YBo4l5iqGkdFa2vTEBvz7oVDOfCbT4/e11AXVNaVtW0/547rDZ/BgvvhnV/D4n/BqD9Dt7NqJkZjolQwN8o9B9wJPAugqj+IyH8AXwliG+B5d1kHd9nPiMgo4F5gmKqW1/FPA4aKyI1AQyBJRPJU9ZiG7nBaV1tmkVv3CSx/Dc64o24lh2B0GwFdh8PKWfDpX5wb7LqNhO3fw2EvU57YfRQhU1WbVyUCQun6GkyCqK+q31b4w/qbymsp0F1EuuIkhgnA5Z4biMgAnIQzxvNylapO9NjmKiCtppMDONOMdmxej/pJQQ16G10KDsIHt0DLnjDsD5GOJrrFxTkjw/ZKd+4o/2IqFOz3vq3dRxGSlJQU9uzZQ4sWLSxJ1CBVZc+ePaSkBDepVjBnvt0i0g13XmoRGQfs8BNQiYhMxhn1NR54UVVXicgDQIaqzgam4tQQ3nI/LFtUNbyjTwVhXXZe7M8BseB+yN0B18y3GdcClZgCQybDgInw9y6RjqZW6dChA1lZWYSjzdD4l5KSQocOHSrf0EMwCeImnOv9J4jINuB/wER/BVR1DjCnwrL7PZ5XOkCOqk4HpgcRZ7UoLi1j4+48RvSK4cbbjf+FZS/BaZOh4ymRjib21GsW6QhqnfI7mU1sCKYX00b3hJ4KnIDToHxGuAKLtE27D1FcqrFbgyjMg9mToXk3GHFfpKOpnT57yOkdZkwtVWmCcO9mvkdEnhSR0UA+MAlYD/wy3AFGytpYb6D+9AHYvxXGPgWJ9SIdTe30xSPweD/471Qo9D1UgjGxKpAaxKs48z78CPwGWAhcClysqmP9FYxlmdl5xAl0S43BBLF5EXz7LAy6DjqfFuloYpu/+yh++yV0GgILH4R/9nNuuivK9769MTEokDaI41T1RAAReR6nYbpT+eRBtVXmzly6tGhASmKMDV5XlA/v3+QM2z3qT5GOJvZV1pX18pmQtQwWPuR0CFj0JAy9AwZebZ0CTMwLJEEUlz9R1VIRyartyQEgc1dubF5eWvgQ7N0IV86GpAaVb2+qrsNA+NU7sHmx8/v/+G74+gnnslORl0tPdg+FiRGBXGI6SUQOuo9coF/5cxE5GO4AI6GguJRNuw/FXgP11qWw5N/Ot9fjhlW+valenU+Dqz50knPTjt6TA9g9FCZmVFqDUNUYu8ZSdRtzDlGmMTYHRHEBvH8jNGrnzAhnIue4YdD1TPhz00hHYkyVxPAtwuGzbpfzzS+mphn9799hdyZc8TakNI50NMbuEja1QDDDfdcZa3fmkhAndGkRI9fwt3/v9KAZcIUzZaiJfkufh7LSSEdhjF+WILzIzM7juNQGJCXEwK+npAjeu8mZf/nshyIdjQnUR7+D50fCtmWRjsQYn2LgDFjzMrNzY6f94cvHYNcquOCfNitatPF3D8UlL8DB7fDcSPjwdsj3MmKsMRFmbRAV5BeVsHVfPuMGBjeoVY2Z2t17L5jZN1vXyWhT2d+j+2hY+DfnpsbV78Pov8BJlzmjyhoTBeyTWMH6XXmoEr3TjPrqImldJ2NPShM492FnVrsWxzu90F46F3aujHRkxgBWgzhGZnYeUEtmkTOxoc2JcPXHsHyGczf2s2dCQhIUHz52W7vJztQgq0FUkJmdS1JCHJ1jpQeTqR3i4uDkX8HNy5yf3pIDWE3R1ChLEBVkZudyfGpD4uOsH7uJgPrN4cLHIx2FMYAliGNk7syN3vYHY4ypQZYgPOQWFLP9QAE9ovkOan9dJ40xphpZI7WHIw3UraI4QYx7AV6+0OlHf+K4SEdjImHHCmh7UqSjMHVAWGsQIjJGRNaKyHoRudvL+jtEZLWI/CAin4pIZ3d5fxFZLCKr3HXjwxlnuXXZMTAG04qZkNQITjg/0pGYcPJVI5Q4eOl82Ph5jYZj6qaw1SBEJB54ChgNZAFLRWS2qq722Ox7IE1V80XkBuARYDzOtKZXquo6EWkHLBOReaq6P1zxgjPNaL3EeNo3jdIpOovynRuq+lxk04jWdr66sh7cDq9dAq+Ng4ufsVqkCatw1iAGAetVdaOqFgEzgZ9NUaqqC1W1fI7GJUAHd3mmqq5zn28HdgGpYYwVgHXZefRo3ZC4aO3B9NNHUJTn3G1r6qbG7eDqudBxELx9LSz+d6QjMrVYOBNEe2Crx+ssd5kv1wJzKy4UkUFAErDBy7rrRCRDRDJycnKqGK5Tg4jqMZhWvA5NOjnzIJu6q15TuOId6HUhzLvHubmurCzSUZlaKCoaqUXkCiANGFZheVvgVWCSqh7zH6Cq04BpAGlpaVqVGPbnF5GTWxi9s8jl7oSNC+GMO2ysHuPMd33pyzDnTmeo99xsGPskxCdWfd++xvuyu7jrnHAmiG1AR4/XHdxlPyMio4B7gWGqWuixvDHwEXCvqi4JY5zA0R5MUTsP9Y9vgZbBSRMiHYmJFnHxcP5j0KgtLHwQDuXAL1+B5Cp+hm28L+MK51fRpUB3EekqIknABGC25wYiMgB4FkhX1V0ey5OAd4FXVHVWGGM8Yq3bgylqx2BaMRPap0HL7pGOxEQTERh2J1z4hFPDfPkCyAvxcqsq7NtUreGZ2Ba2GoSqlojIZGAeEA+8qKqrROQBIENVZwNTgYbAW+JM0bhFVdOBXwJnAi1E5Cp3l1ep6vJwxbsuO5dGyQm0bZISrkOEbuePkL0Szns00pGYaDVwkjNp1FtXwYtnO20Uzbv6L1NWBrvXwuavYfNi2LwIcrfXSLgmNoS1DUJV5wBzKiy73+O51/kxVfU14LVwxlbR2p25dG/dEInGuYRXzIS4ROjzi0hHYqJZz3Phytnw4jnwRP9j1zdoBZe/4SSCzYtgy2I47E5U1KgtdB4CnU6DOb+v0bBN9IqKRupIU1Uys3MZ07dNpEM5VmmJ0/7Q4xxo0CLS0Zho12kw4KO/xqFd8NxZzvPmx0HP85yk0Pk0aNbVuVwFliDMEZYggN15RezLL6Z7NA6xsfFzyMu2xmlTPca96HSTbtzW9zYNWvnuxWTqFEsQRPkQGz/MhJSm0P3sSEdiaoO+l1S+TcWurB/eDsumwxVvhyUkE72sQz1HezBFXRfXgoOw5kPnnzohOdLRmLpq5P1Qrzl89Du7Ia+OqdM1iLQHF7A7r+jI60EPfQpAy4ZJZNw3OlJhHbVmNpQctqE1TGTVawZn/wXeuwGWvwYnXxnpiEwNqdM1CM/kEMjyGrdiJjTvBh3SIh2JiSXhmDPkpMuctosFf4L8vaHvx8SUOl2DiGr7t8CmL+Gse4/2LjEmEOEYDkPEuWv7mTPgkymQ/kT1H8OG+Ig6dboGEdV+eNP52e+XkY3DmHKte8OpN8B3L8PWpdW/fxviI+pYgohGqs7lpU5DoFmXSEdjzFHD74ZG7eCj2517dEytZgkiGm3/Dvass3sfTPRJbgRj/uYM/7L0+UhHY8KsTieIlg2TglpeY1bMhPhkZ+Y4Y6JN77HQbSQsfMgZhr465NllpGhUpxupo6Ira0UlRfDjLGfO6ZQmkY7GmGOJwHlT4d+nwvz74JIq1iT2bYJXL/a/zU9z4ITzqnYcX6xx3Kc6XYOISus/cQZQs8tLJpq16AZn3O6ME7bxv6HvJ3s1vHCO03W2XjPv28QlwMzL4ct/OO1z1c0ax32q0zWIqLTidWiQCt1GRDoSY/w743b44Q3nDusbFkFCkJdmt3wD/7kUEuvDNR9Dq17etys+DO/fBJ/+GXatgfR/OTPqmbCzGkQ0ObwPMj+GEy+tnqkjjQmnxHrOHCV71sHifwVXdt0CeGUs1G8B18zznRzKj3PJCzDiPvjxTZh+XvW1fRi/LEFEk1XvQmkR9Bsf6UiMCUz30dDrQvjvVOfmzkD8OAten+DMjnjNPGjWufIyInDmnTD+Ndj1E0w7C7Z/X7XY92+Fj++p2j5qOUsQ0WTFTEjtBW1PinQkxgRuzMPOCXzu3ZVv+800ePvX0PFUuOpDZxa8YPS6EK6d58zH/eK5sDKEEWZ3roR3roPHT4JvpwVfvg6xBBEt9myArd84jdM2tIaJJU06wLA/wNqPYO1c79uowsK/wdw7nYmKrng79F56bU6E3yx0vkjNugY+e6jyUWZV4X9fwGuXwDOnO6MkD74eblnue4yq5Cgc/r+GWSN1tPjhDUCc9gdjYs2pNzodLObeBV2HQVL9o+vKypzlS5+D/hPhwicgvoqnnoapMGk2fHgHfPEIfP04lBYeu12DVnDu3531O5Y7HUBG/BFOufZor6mKXVnLyuDViyArw/ni1qJb1WKNYWGtQYjIGBFZKyLrReSY+qeI3CEiq0XkBxH5VEQ6e6ybJCLr3MekcMYZceVDaxw3DJq0j3Q0xgQvIckZzG//FvjqH0eXlxTBO79xksOQm2HsU1VPDkeOmQxjn4Rz/uo9OYDTVXXW1VCYCxf8E25bCWf+3neXWoC4OLjoaSfOd39bp4cUEQ1Hv2JAROKBTGA0kAUsBS5T1dUe25wFfKOq+SJyAzBcVceLSHMgA0jDmWB3GTBQVff5Ol5aWppmZGSE5b2E3ebF8NIYuPhZu//BxLYHW0NJwbHLkxrA/9sevuNO8XO56pevOjeexsUHt88fZ8Hb18JZ98GwO6sWXxQTkWWq6nVOgXDWIAYB61V1o6oWATOBsZ4bqOpCVc13Xy4BOrjPzwEWqOpeNyksAMaEMdbIWvE6JDaAEy6IdCTGVI235ABQdKhm4/DUOz345ABw4jjoOw7++zBs+67644oB4UwQ7YGtHq+z3GW+XAuUt3AFVFZErhORDBHJyMnJqWK4EVJcAKvec3pnJEfZlKfG1HXnPwoNWzu9noryK9++lomKXkwicgXO5aSpwZRT1WmqmqaqaampqeEJLtwy50LhAbu0ZEw0qtcMLvq3czPggvsjHU2NC2cvpm1AR4/XHdxlPyMio4B7gWGqWuhRdniFsp+HJcpI8DY42KsX2eBgxoSqQSvfA+5V1XHD4dSbYMlT0GMMdB9V9X3GiHAmiKVAdxHpinPCnwBc7rmBiAwAngXGqKrnX3ce8FcRKe9qcDZQe255tMHBjKle4f5iNfJ+2PCZMybUjYuhfvPwHi9KhO0Sk6qWAJNxTvZrgDdVdZWIPCAi6e5mU4GGwFsislxEZrtl9wJ/wUkyS4EH3GXGmGjm6xt7dXyTj6TEFPjFNMjfAx/cGp5RZaNQ2Lq51rSY6OZaVgabv4KXL/S9zZQDNRePMSY4X/0ffDIFLnoG+l8W6Wiqhb9urnYndU3IyYQfZsIPb8KBrZVvb4yJTkNugcz5MOdO6DwksIEGY1hU9GKqlQ7tcQYmm3YWPHWK880j9QRn2GJjTGyKi4eLn3Gev3cDlJVGNp4wsxpEqHxNU5jcBLqcAevmQVmJM7DY2Q85Yyw1au1s8/E94etxYYwJr2ad4bxHnASx+Ek4/dZIRxQ2liBC5avHUeEB2JYBp94A/SZAm77HbmNdWY2JbSdd5oxc++lfnNkf25wY6YjCwhJEONy+uvoGJDPGRB8RZ/C/rd84d1n/ZmGtnAbVzmLhYMnBmNqvQQsY+2+YcQk81NrL+ti/8dUaqY0xJlT+7qquBTe+WoIwxhjjlSWIUNXWO0aNMdVn3QIoiN2bX+1ieahi/NqiMaYGzBgHEget+0Ln06HzadBpiDNlKvjuLh8l7ReWIIwxJlyunA1bFsPmr2HZdPjmaWd5yx7Q6bSoH7jTEoQxxlSFv6HGjxvmPMCZn3vHCidZbF7kTBQW5SxBGGNMVQR6KSghCTqe4jzOuM0ZpuOB6B423BqpjTEmEkKZJ7uGWYIwxphotObDSEdgCcIYYyLGV7f4uAR4YyJ88ueIjhhrbRDGGBMpvtovigtg7l3w1T9g+/fONAENWtRsbFgNwhhjok9iCqQ/Aen/cno8TRvmJIoaZgnCGGOi1clXwjUfO89fOAe+e7VGDx/WBCEiY0RkrYisF5G7vaw/U0S+E5ESERlXYd0jIrJKRNaIyBMiIuGM1RhjolL7k+G6/zp3Yc+eDB/cCiWFNXLosCUIEYkHngLOBXoDl4lI7wqbbQGuAv5ToewQ4HSgH9AXOAUYFq5YjTEmqjVoAVe8A2fc7tyR/dK5cCAr7IcNZyP1IGC9qm4EEJGZwFhgdfkGqrrJXVdWoawCKUASIEAikB3GWI0xJrrFxcOoKdB+ILx7A/xfX5xTZQXVOI5TOC8xtQe2erzOcpdVSlUXAwuBHe5jnqquqfYIjTEm1vS6EH7zGV6TA1TrOE5R2UgtIscDvYAOOEllhIgM9bLddSKSISIZOTk5NR2mMcZERmqPGjlMOBPENqCjx+sO7rJAXAwsUdU8Vc0D5gKnVdxIVaepapqqpqWmplY5YGOMMUeFM0EsBbqLSFcRSQImALMDLLsFGCYiCSKSiNNAbZeYjDGmBoUtQahqCTAZmIdzcn9TVVeJyAMikg4gIqeISBZwKfCsiKxyi88CNgA/AiuAFar6QbhiNcYYc6ywDrWhqnOAORWW3e/xfCnOpaeK5UqB34YzNmOMiWn+5qGoJjYWkzHGxKIamJI0KnsxGWOMiTxLEMYYY7yyBGGMMcYrSxDGGGO8sgRhjDHGK1H1MZ5HjBGRHGBzFXbREthdh8pG8tixWDaSx7b3HBtlI3nsqpTtrKreh6JQVXs4STKjLpWN1bjt92XvOVrLxnLcvh52ickYY4xXliCMMcZ4ZQniqGl1rGwkjx2LZSN5bHvPsVE2kseuatxe1ZpGamOMMdXLahDGGGO8sgRhjDHGqzqfIERkjIisFZH1InJ3kGVfFJFdIrIyhON2FJGFIrJaRFaJyK1BlE0RkW9FZIVb9s8hHD9eRL4XkQ+DLLdJRH4UkeUikhHCcZuKyCwR+UlE1ojIMTMF+ijX0z1m+eOgiNwWxHFvd39XK0XkdRFJCaLsrW65VYEc09vnQkSai8gCEVnn/mwWRNlL3WOXiUhakMed6v6ufxCRd0WkaRBl/+KWWy4i80WkXTDH9lj3OxFREWkZxLGniMg2j7/3ecEcV0Rudt/3KhF5JIjjvuFxzE0isjyIsv1FZEn5/4aIDPJW1k/5k0Rksfv/9YGINPZSzut5I9DPV9DC0Xc2Vh5APM7ERMcBSTiTE/UOovyZwMnAyhCO3RY42X3eCMgM9NiAAA3d54nAN8CpQR7/DuA/wIdBltsEtKzC7/xl4Nfu8ySgaYh/t504N/gEsn174H9APff1m8BVAZbtC6wE6uMMj/8JcHywnwvgEeBu9/ndwN+DKNsL6Al8DqQFedyzgQT3+d+DPG5jj+e3AM8Ec2x3eUecScM2+/rc+Dj2FOD3Afx9vJU9y/07JbuvWwUTs8f6x4D7gzjufOBc9/l5wOdBxr0UGOY+vwb4i5dyXs8bgX6+gn3U9RrEIGC9qm5U1SJgJjA20MKq+gWwN5QDq+oOVf3OfZ6LM+te+wDLqjpzdYOTIBKBgHsbiEgH4Hzg+aCCriIRaYLzj/ECgKoWqer+EHY1EtigqsHcOZ8A1BORBJyT/fYAy/UCvlHVfHVmSfwv8At/BXx8LsbiJEfcnxcFWlZV16jq2soC9VF2vhs3wBK8TNDlp+xBj5cN8PMZ8/O/8H/AXSGWrZSPsjcAD6tqobuNl1l1/B9XRAT4JfB6EGUVKP/W3wQ/nzEf5XsAX7jPFwCXeCnn67wR0OcrWHU9QbQHtnq8ziLAk3R1EpEuwACcmkCgZeLd6u8uYIGqBlwW+CfOP21ZEGXKKTBfRJaJyHVBlu0K5AAvuZe3nheRBiHEMAEf/7jeqOo24FGcuc53AAdUdX6AxVcCQ0WkhYjUx/lm2DHIeAFaq+oO9/lOoHUI+6iqa4C5wRQQkYdEZCswEbi/su0rlB0LbFPVFcGU8zDZvcT1YpCXTHrg/M2+EZH/isgpIRx7KJCtqsHMynMbMNX9fT0K3BPkMVdx9AvqpVTyOatw3gjL56uuJ4iIE5GGwNvAbRW+sfmlqqWq2h/nG+EgEekb4PEuAHap6rJQ4gXOUNWTgXOBm0TkzCDKJuBUq59W1QHAIZzqcMBEJAlIB94KokwznH+8rkA7oIGIXBFIWVVdg3NpZj7wMbAcKA0mZi/7VIKo8VUHEbkXKAFmBFNOVe9V1Y5uuclBHK8+8P8IMql4eBroBvTHSeqPBVE2AWgOnArcCbzp1giCcRlBfAlx3QDc7v6+bsetKQfhGuBGEVmGc/moyNeG/s4b1fn5qusJYhs/z9Id3GU1QkQScf7IM1T1nVD24V6iWQiMCbDI6UC6iGzCuaQ2QkReC+J429yfu4B3cS7TBSoLyPKo7czCSRjBOBf4TlWzgygzCvifquaoajHwDjAk0MKq+oKqDlTVM4F9ONd9g5UtIm0B3J9eL3uEg4hcBVwATHRPHqGYgZdLHn50w0nIK9zPWgfgOxFpE0hhVc12vwSVAc8R/OfsHfdS7Lc4NWWvDeTeuJchfwG8EcQxASbhfLbA+QITTMyo6k+qeraqDsRJTht8xOftvBGWz1ddTxBLge4i0tX9ZjoBmF0TB3a/0bwArFHVfwRZNrW8N4qI1ANGAz8FUlZV71HVDqraBef9fqaqAX2bFpEGItKo/DlOA2jAPbhUdSewVUR6uotGAqsDLe8K5ZvdFuBUEanv/t5H4ly7DYiItHJ/dsI5cfwnyOOD87ma5D6fBLwfwj6CJiJjcC4npqtqfpBlu3u8HEuAnzEAVf1RVVupahf3s5aF07i6M8Bjt/V4eTFBfM6A93AaqhGRHjidIYIZ6XQU8JOqZgVRBpw2h2Hu8xFAUJNGe3zO4oD7gGe8bOPrvBGez1d1tHTH8gPnmnImTra+N8iyr+NUf4tx/gGuDaLsGTjVwB9wLlssB84LsGw/4Hu37Ep89LQIYD/DCaIXE05vrxXuY1Wwvy93H/2BDDf294BmQZRtAOwBmoRw3D/jnOBWAq/i9nAJsOyXOIlsBTAylM8F0AL4FOek8QnQPIiyF7vPC4FsYF4QZdfjtLOVf8a89kTyUfZt9/f1A/AB0D7U/wX89H7zcexXgR/dY88G2gZRNgl4zY39O2BEMDED04HrQ/gbnwEscz8n3wADgyx/K865KBN4GHekiwrlvJ43Av18BfuwoTaMMcZ4VdcvMRljjPHBEoQxxhivLEEYY4zxyhKEMcYYryxBGGOM8coShDE+iEip/HwE2bvd5Z+LMwLwChH5uvy+DhFJEpF/ijMy8DoRed8d96p8f21EZKaIbHCHKpkjIj1EpIscOxrpFBH5vfv8VHfYiOXijIA7pQZ/DaYOS4h0AMZEscPqDGfizURVzXDHo5qKM/zHX3GGSOipqqUicjXwjogMdsu8C7ysqhPAGd4ZZ8ycrcfu/mdeBn6pqitEJB5nZFdjws4ShDFV8wVwmzv20NVAV1UtBVDVl0TkGpy7ahUoVtUjd8eqO4idO+iaP61wbqrC3Xewd58bExJLEMb4Vk9+PmHM31S14vg8F+Lc8Xs8sEWPHXAxA+jjPvc3QGK3CsdqgzMiKDhDZq8Vkc9xBgx8WVULAn0TxoTKEoQxvvm7xDRDRA7jDCFxM1DVGbw2eB7Ls51BVR8QkRk4Y19djjMe1fAqHs+YSlmCMCY0E1X1yJSrIrIX6CQijdSZyKXcQKB8WtdxoR5MVTcAT4vIc0COiLRQ1T2h7s+YQFgvJmOqgaoewmlM/ofbkIyIXIkze91n7iPZc5IlEeknIkMr27eInO8xn0F3nPko9lfvOzDmWJYgjPGtXoVurg9Xsv09QAGQKSLrcGYFu1hdOKOyjnK7ua4C/oYz+1dlfoXTBrEcZ5TTieUN4caEk43maowxxiurQRhjjPHKEoQxxhivLEEYY4zxyhKEMcYYryxBGGOM8coShDHGGK8sQRhjjPHq/wNFIswiUzm/jwAAAABJRU5ErkJggg==\\n\",\n      \"text/plain\": [\n       \"<Figure size 432x288 with 1 Axes>\"\n      ]\n     },\n     \"metadata\": {\n      \"needs_background\": \"light\"\n     },\n     \"output_type\": \"display_data\"\n    },\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 295 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"value1 = ft_recall_5\\n\",\n    \"value2 = cs_recall_5\\n\",\n    \"\\n\",\n    \"length = len(value1)\\n\",\n    \"plt.plot(range(length), value1, \\\"-s\\\", label=\\\"Finetune\\\")\\n\",\n    \"plt.plot(range(1, length), value2, \\\"-s\\\", label=\\\"w/o Pretrain\\\")\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"plt.xticks(range(length))\\n\",\n    \"plt.ylabel('Recall@5')\\n\",\n    \"plt.xlabel(\\\"EPOCHS\\\")\\n\",\n    \"plt.legend()\\n\",\n    \"plt.show()\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 152,\n   \"id\": \"confirmed-exhibit\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"image/png\": \"iVBORw0KGgoAAAANSUhEUgAAAY4AAAEGCAYAAABy53LJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA7NElEQVR4nO3dd3wUZf7A8c83nRJAgSBVEEIXEaIURVGK2CKnoCAqlhMLqGc78fQ87uQ8+53tJ4oiFhQ5VOROFFBAUaQkCkpvggaB0Dup398fM+HWsLvZSbLZlO/79dpXZmfmmXkm2cx3nzLPI6qKMcYYE6qoSGfAGGNMxWKBwxhjjCcWOIwxxnhigcMYY4wnFjiMMcZ4EhPpDJSFevXqafPmzSOdDWOMqVDS09N3qmr9wuurROBo3rw5aWlpkc6GMcZUKCKy2d96q6oyxhjjiQUOY4wxnljgMMYY44kFDmOMMZ5Y4DDGGONJWHtVicgA4DkgGnhNVR8PsN8VwFTgDFVNE5F+wONAHJAN3K+qc9x95wENgSNu8v6qmhnO6zDGmLKUMnY2Ow9mH7e+Xs040h7uF7a0oQpb4BCRaOAloB+QASwRkemqurLQfonAXcAin9U7gUtV9VcR6QjMBBr7bB+mqta/1pgqoKQ3wrK4kZY2f/kNtr600oYqnCWOM4H1qroRQEQmA5cBKwvt9yjwBHB/wQpV/d5n+wqgmojEq2pWGPNrTLkXyZtopNKW9EZYkvRlfc0Hs3LJ3H806HFfmrueKBGiBKJEEPdnlEB0lARNW1rCGTgaA7/4vM8AuvnuICJdgKaq+omI3I9/VwDfFQoab4hIHvABMFb9TCoiIiOAEQDNmjUr/lUYU44U5yaoquTkKdl5+UHT5+crUUFuPGX5LTgvXzmcncuR7Lygx527JhNVJS8f8lVRVfLVWc5X59qDOZiVS424aET8X3e4rvnVrzawfX8WmQeyyNx/9NjPQ0VcL8BTM9cUuU+4RezJcRGJAp4Frg+yTwec0kh/n9XDVHWLW8X1AXAt8FbhtKr6KvAqQEpKis1WZSo0VWVbEd9Ez3t6Htm5+eTk5ZOdl09Obv6xgBGKU/40g+goITZaiIuOIi4milifn8EMn7CY6CjnW6/4fBuOEgkajAAuem4+R3PyOJydx5GcPI5k54Wc5xveWBLSfoF0/MtMoqOEWgkx1K4WS+1qsdRyf9auFhs07ZjpKziSncdhN89HcnKda3CvI5jHZqymWmw0DWrFk5SYQIdGtTivTRJJteJpUCueu99fFjDtmrEDUJ/gmK+K5kOeKvmqpIz9vFi/Cy/CGTi2AE193jdx1xVIBDoC89xofxIwXURS3QbyJsBHwHWquqEgkapucX8eEJF3carEjgscxoRTuKowFv+pL7/uO8K6zIOs336QtdsPOMuZBzmYlRv0uKc2ru3e6J0bf2x0FLExUT5BQHhsxuqA6e/u25rsvDwn2BQEIPdnTp6yPvNgwLR7D2cf900/X5W8fKWoSUYb1UkgITaa6nHRVIuNplpcjM9yNA9PWx4w7Ue39/QJUD7ByieAnf/MlwHT/+mituw7kuO+co8tZ+w5wr4jOUHz/eF3GVSPi6GaT15rxsdQr2Y81eOi2bzrcMC0P47pT834mIAlnWCBIz4mOmi+ykI4A8cSIFlEWuAEjCHA1QUbVXUfUK/gvdtb6j43aNQBPgFGq+o3PvvEAHVUdaeIxAKXAOEPr8YUEqwaYvFPu4udtuOYmRz2qa6oVzOe5KSaXN6lMckNEvlzkJvo80NPLzLfwQLHXX2Tg6b9ZPQnAbd9POrsoGmbB0n72vAzgqYNFjhOb3ZC0LRFGXFOy6Dbg+X7hzEXBE378dJfA25LTAhemqlXMy7gl4uilCRtqMIWOFQ1V0RG4fSIigYmqOoKEfkbkKaq04MkHwW0Ah4RkUfcdf2BQ8BMN2hE4wSN8eG6BmP82Xc4+DfRK1/5ttjHvjKlKckNapKclEhyUk1OqPHbf/ZggaOyKumNsCxupKWtJL29yqKnWFjbOFR1BjCj0LpHAuzb22d5LDA2wGG7llb+jAnV0Zw8vliVycdLtzBvzY6g+076fbeg24e9tijgtjGpHYKmjeRNNFJpS3ojLEn68v7NP1KkqF4HlUFKSorasOrGq7x85dsNu5i2dAszl2/jQFYuSYnxpJ7WiNe+/ilguk2PXxz0uMGqP4pKa0xZEpF0VU0pvL5KzMdhjD+BGqnrVIvl8i5N+M8Pv7LjQBaJ8TEM6HgSA09vTPdT6hIdJUEDhzGVnQUOU2UFaqTeeySHdxZupneb+gw8vTHnt00iIfa3PVmsCsNUZRY4TIUWarfY3Lx8th/IYuveI2zZe4Rf9wZ/JmLJQ32pXT1wz5fy3nhpTDhZ4DAVWrCuraPe/Y6t+47y694jbN9/lHwPzXnBgoYxVZ0FDhNRXh6kU1V2HsxmXeYB1mceZN32wA+kASzfso+GtavRs2U9GtVJoFGdas6rdgIN61Sj419mluq1GFNVWOAwERWsxPDN+p2s236Ate5T1OsyD7DH5xmKxITgH995959Xqnk1xjgscJhyq+B5h9rVYmndoCYDOjakdcHDcQ1qkpQYT4sHZxRxlMCskdqY4rHAYcrU0Zw8fsjYR9rm3aRv2hN033dv7kZyUiL1asYFHNOnJKyR2pjiscBhSixYO8XMP5xD2uY9pG/eQ9qm3fy4ZR85eU4rdcv6NYIet2fLekG3F5zDSg3GlC0LHKbEgrVTdHWHeI6LjqJTk9rceHYLUk4+ka4nn8CJNeKCPkUdCis1GFP2LHCYEsk8EPx5iAcvbEtK8xPo2Li23+GgrcRgTMVjgcN4cjQnj8U/7Wb+uh3MX7eT1dsOBN3/lnODD1ttJQZjKh4LHAYI3k7x1o3djgWKxZt2k52bT1x0FCnNT+CBAW154rPAczwYYyofCxwGCN5OcdHz8wFo3aAm13Y/mV7J9ejWoi7V4pyqJwscxlQtFjgM+UWMxfH04NM4u1U9Tqqd4He7tVMYU7VY4Kiijubk8e2GXcxetZ0vVm0Puu+grk2Cbrd2CmOqFgsclUhR4z7tPpTNnNWZzF65jfnrdnI4O4/qcdGc27o+ny7fFoEcG2MqIgsclUiwdorB4xaQvnkP+Qon1Urg8i6N6duuAd1PqUtCbHSJn6cwxlQdYQ0cIjIAeA6IBl5T1ccD7HcFMBU4Q1XT3HUPAjcBecCdqjrTyzHNbx3KymPU+cn0a9eAjo1rHTeEh7VTGGNCFbbAISLRwEtAPyADWCIi01V1ZaH9EoG7gEU+69oDQ4AOQCPgcxFp7W4u8pjmeDPu6hV0u7VTGGNCFRXGY58JrFfVjaqaDUwGLvOz36PAE4DvI8iXAZNVNUtVfwLWu8cL9ZhVyqKNuxj66sJIZ8MYU0WEM3A0Bn7xeZ/hrjtGRLoATVW1cAV7oLRFHtPn2CNEJE1E0nbs2FG8KyjnlmzazdXjF3LVqwtZlxl8UiNjjCkt4QwcQYlIFPAscG84jq+qr6pqiqqm1K9fPxyniJj0zbu55rVFDB73LWu3H+Dhi9sx/4/nBWyPsHYKY0xpCmfj+Bagqc/7Ju66AolAR2Ce21B7EjBdRFKLSBvsmJVa+uY9/Ovztcxft5O6NeJ46KJ2XNP95GNPcFs7hTGmLIQzcCwBkkWkBc7NfQhwdcFGVd0HHJtwQUTmAfepapqIHAHeFZFncRrHk4HFgAQ7ZkUX6DmMOtVi6dS0Dl+t3cGJNeJ48MK2XNvjZKrHWW9qY0zZC9udR1VzRWQUMBOn6+wEVV0hIn8D0lR1epC0K0RkCrASyAVGqmoegL9jhusaylqg5zD2Hsnhx4y9PDCgLdf1OJka8RYwjDGRI6rBxymqDFJSUjQtLS3S2ShSsIfwVvz1AgsYxpgyJSLpqppSeH3EGseNNxY0jDHlhQUOY4wxnljgKCcy9hyOdBaMMSYkFjjKgZ93HeaqVxYiAbbbcxjGmPLEKs4jbNPOQwwdv5DD2Xn8546z6di4dqSzZIwxQVngiKCNOw4ydPxCsnPzee/m7rRvVCvSWTLGmCJZ4IiQ9ZkHGDp+Efn5ynsjutP2JAsaxpiKwQJHBKzZdoBhry0EhMkjupPcIDHSWTLGmJBZ43gZW7V1P0PHLyRKLGgYYyomK3GUoeVb9nHN64uoFhvNuzd3p0W9GpHOkjHGeGYljjLyQ8Zerh6/kBpxMbw/oocFDWNMhWUljjLw/c97uG7CYmpXi+W9m7vT9MTqkc6SMcYUmwWOUhZoaPQogc/+cA6N61SLQK6MMab0WFVVKQs0NHq+YkHDGFMpWOAwxhjjiQUOY4wxnljgMMYY44kFDmOMMZ6ENXCIyAARWSMi60VktJ/tt4rIjyKyVES+FpH27vph7rqCV76IdHa3zXOPWbAtKZzX4FWgIdBtaHRjTGURtjnHRSQaWAv0AzKAJcBQVV3ps08tVd3vLqcCt6vqgELHORWYpqot3ffzgPtUNeRJxMtyznFVpcc/5tC5aR3GXdu1TM5pjDHhEIk5x88E1qvqRlXNBiYDl/nuUBA0XDUAf1FsqJu2Qvhxyz627T9Kv/YNIp0VY4wJi3A+ANgY+MXnfQbQrfBOIjISuAeIA873c5yrKBRwgDdEJA/4ABirfopNIjICGAHQrFmz4uS/WGat2E50lHB+23JVg2aMMaUm4o3jqvqSWw31APCw7zYR6QYcVtXlPquHqeqpQC/3dW2A476qqimqmlK/fv0w5f54s1Zu44zmJ3BCDWvTMMZUTuEMHFuApj7vm7jrApkMDCy0bgjwnu8KVd3i/jwAvItTJVYubNp5iLXbD9K//UmRzooxxoRNOAPHEiBZRFqISBxOEJjuu4OIJPu8vRhY57MtCrgSn/YNEYkRkXrucixwCeBbGomo2Su3A1j7hjGmUgtbG4eq5orIKGAmEA1MUNUVIvI3IE1VpwOjRKQvkAPsAYb7HOIc4BdV3eizLh6Y6QaNaOBzYHy4rsGr2Su3065hLRv91hhTqYV1dFxVnQHMKLTuEZ/lu4KknQd0L7TuEFAu+7juPJhF2ubd3HF+ctE7G2NMBRbxxvHKYs6qTPLVqqmMMZWfBY5SMmvldhrXqUaHRrUinRVjjAkrCxyl4HB2LvPX7aBf+waISKSzY4wxYWWBoxTMX7eTrNx8+ls1lTGmCrDAUQpmrdhO7WqxnNHixEhnxRhjws4CRwnl5uUzZ/V2zm+bRGy0/TqNMZWf3elKKG3zHvYczrFqKmNMlWGBo4RmrdhOXEwU57Quu/GwjDEmkixwlICqMnvVNs5uVY8a8WF9ltIYY8oNCxwlsHrbAX7ZfcSqqYwxVYoFjhKYvXI7ItCnnQUOY0zVYYGjBGat3EaXZidQPzE+0lkxxpgyY4GjmLbsPcLyLfutmsoYU+VY4Cimz23uDWNMFWWBo5hmrdxGq6SanFK/ZqSzYowxZarIwOHOuneLiHwmIj+4r09F5FZ3QqUqZ9/hHBZt3G2lDWNMlRTKwwdvA3uBMUCGu64Jzmx97wBXhSNj5dncNZnk5qu1bxhjqqRQAkdXVW1daF0GsFBE1oYhT+XerJXbSEqM57QmdSKdFWOMKXOhtHHsFpHBInJsXxGJEpGrcOYJr1KO5uTx5Zod9G3fgKgom3vDGFP1hBI4hgCDgO0istYtZWwDLne3BSQiA0RkjYisF5HRfrbfKiI/ishSEflaRNq765uLyBF3/VIRGeeTpqubZr2IPC9lPHPStxt2cSg7z6qpjDFVVpFVVaq6CbcdQ0Tquut2FZVORKKBl4B+OFVbS0Rkuqqu9NntXVUd5+6fCjwLDHC3bVDVzn4O/TJwM7AImOHu/2lR+Skts1Zup2Z8DD1a1i2rUxpjTLniqTuuqu7yDRoi0i/I7mcC61V1o6pmA5OBywodb7/P2xqABju/iDQEaqnqQlVV4C1goJdrKIn8fOXzVds5t0194mOiy+q0xhhTrpR0SNfXgWYBtjUGfvF5nwF0K7yTiIwE7gHigPN9NrUQke+B/cDDqjrfPWaGzz4Z7rrjiMgIYARAs2aBsujN0oy97DiQZdVUxoRBTk4OGRkZHD16NNJZqXISEhJo0qQJsbGhPWFRZOAQkemBNgElrq9R1ZeAl0TkauBhnG6+W4FmqrpLRLoC00Skg8fjvgq8CpCSkhK0JBOqWSu2ExMl9G6TVBqHM8b4yMjIIDExkebNm1PGTZdVmqqya9cuMjIyaNGiRUhpQilx9AKuAQ4WWi841VGBbAGa+rxv4q4LZDJO+wWqmgVkucvpIrIBaO2mb+LhmKVq1spt9GhZl9rVquRzj8aE1dGjRy1oRICIULduXXbs2BFymlACx0LgsKp+6eeEa4KkWwIki0gLnJv7EODqQumTVXWd+/ZiYJ27vj6wW1XzROQUIBnYqKq7RWS/iHTHaRy/DnghhGsosfWZB9m44xDX92xeFqczpkqyoBEZXn/vRTaOq+qFqjo3wLZzgqTLBUYBM4FVwBRVXSEif3N7UAGMEpEVIrIUp51juLv+HOAHd/1U4FZV3e1uux14DVgPbKCMelTNdgc17GtzbxhTaUVHR9O5c+djr02bNtGzZ89iH2/ixIn8+uuvpZjD8sFz47jbJXePquYXta+qzsDpMuu77hGf5bsCpPsA+CDAtjSgo5c8l4bZK7dxauPaNKpTraxPbYwpJGXsbHYezD5ufb2acaQ9HKyzZ3DVqlVj6dKlv1m3YMGCYh9v4sSJdOzYkUaNGhX7GOVRSN1xReQEEXlRRL7EeTbjUxGZICI1wpu98iFz/1G+/2Wv9aYKh6eSYUzt419PJUc6Z6Yc8xc0gq0viZo1nRGw582bR+/evRk0aBBt27Zl2LBhOE8FQHp6Oueeey5du3blggsuYOvWrUydOpW0tDSGDRtG586dOXLkCM2bN2fnzp0ApKWl0bt3bwDGjBnDjTfeSO/evTnllFN4/vnnj53/nXfe4cwzz6Rz587ccsst5OXllfo1ehVKr6o6OKWGP6nqKJ/15wGPi8gUYIVPVVKl8/mqTFShf4eTIp2VyudQprf1pkr4639WsPLX/UXv6MdVr3zrd337RrX4y6XBO2ceOXKEzp07A9CiRQs++uij32z//vvvWbFiBY0aNeKss87im2++oVu3btxxxx18/PHH1K9fn/fff5+HHnqICRMm8OKLL/L000+TkpJSZL5Xr17N3LlzOXDgAG3atOG2225j/fr1vP/++3zzzTfExsZy++23M2nSJK677rrQfhlhEkpV1Z+Bp1V1roi8DXQHdgL1gB9xelc9jNNGUSnNXrmNZidWp3UDm3vDmMrMX1WVrzPPPJMmTZyOnQVtIHXq1GH58uX06+dUkeXl5dGwYUPP57744ouJj48nPj6epKQktm/fzhdffEF6ejpnnHEG4AS2pKTIPw4QSuA4R1XvdZezgKGqmiYiXYDbgK+B58KVwUg7mJXLN+t3cV2Pk63HR2nIzYJtP0JGGmxJi3RuTDlVVMmg+ehPAm57/5YepZ2dY+Lj448tR0dHk5ubi6rSoUMHvv3Wf0nHV0xMDPn5TvNw4QcdAx17+PDh/OMf/yilKygdoQSOBBERd4iPLsAyd/1yoIuq5lfmG+pXa3eQnZdvkzYF81Sy/6qlGklw42dukEh3AsW2HyHPrYdOLKLBcNEr0GU4xCaUfp6NKSVt2rRhx44dfPvtt/To0YOcnBzWrl1Lhw4dSExM5MCBA8f2bd68Oenp6Vx44YV88IHf/j+/0adPHy677DLuvvtukpKS2L17NwcOHODkk08O5yUVKZTG8cVAH3f5/4BZIvIYTjfbV0TkDGBFmPIXcbNWbOPEGnF0PfmESGel/ArWTvFCF/hoBHz/DsRUg+63wZVvwz2r4N5VwY/76R+d9GkTILf0Gz1NxVWvZpyn9eEUFxfH1KlTeeCBBzjttNPo3LnzsZ5Y119/PbfeeuuxxvG//OUv3HXXXaSkpBAdXfR4d+3bt2fs2LH079+fTp060a9fP7Zu3RruSyqSFPQKCLiD8wDeFOBiVd0uIvWAU4CNOIFnOjBcVYM9DBhRKSkpmpYWerVIuLr6VVpjagfedulz0DgF6reFaD8F3GCllSvGw5y/Q8ZiqNMMzn0AOg3xfxxT4a1atYp27dpFOhtVlr/fv4ikq+pxLfuhDKu+0R2IcLqIzMJ5kjwPuMh93Vueg0ZxlGVXvwpv67Lg27teH3z7/euCb29xLqz/HOaMhY9Hwvxnofdo6HgFRNkIxcZEQkhf3VR1kYj0wKmyOs1dvRAY6z4hbqqaQ7tgzqOQPjG85xGB5H7Qqi+smQFzH4MPb4b5zzgBZMYfA5dYigpKxphiCbnM7z4pPtt9maoqL9dpc5g7FrIOQrdbYdHL4T+vCLS9GFpfCKs+hrn/gH9fH3h/ew7EmLAJ5QHAAzgTLAm/nWhJAFXVWmHKmylvfvoKPn0AMlc6VUgXPgFJ7WD5B4G/9Ze2qCjo8Dtol+qc98ObS/8cxpigQmnjSCyLjJhybO/PMOvPsHKa00h95dvQ7lKnFACRqRKKioZOV1rgMCYCQilxnBhse2UcaqRezbiAvaoqrUC9m2JrQMF4luc9BD3vgNgKMtCj6v+CmzGm1ITSxpHO/6qqClOcrrmVSpXschuoTSDnkFM11O9RqNPU/z7l1ZuXwsXPQv3Wkc6JqUQef/xxmjZtyrBhw4rcd+LEidx///00btyY7Oxs7r77bm6+OfRS8rRp02jdujXt27f3lMfp06ezcuVKRo8e7SldqEKpqgptLkFTeQ2eGOkcBFYjyX/Qi0uEbT/Ayz3h7D9Ar3srTknJFC3Y8z9hrjqdOXMmU6ZMCXn/q666ihdffJHMzEw6dOhAamoqDRr8bySK3NxcYmL834qnTZvGJZdc4jdwBEuXmppKamqq322lwdOTVCJyAs5sfMfGgFDVr0o7U8aELNhN4uAOmPUQfPUU/DgVLn4GWvUJvL+pOMIwqvJTTz1FfHw8d955J3fffTfLli1jzpw5zJkzh9dff51Jkyaxf/9+srOzqV+/Pps2beLGG29k586d1K9fnzfeeINmzZoFPH5SUhItW7Zk8+bNPPDAAyQkJPD9999z1llnMXLkSEaOHMmOHTuoXr0648ePZ/fu3UyfPp0vv/ySsWPH8sEHH3DTTTfRuXNnvv76a4YOHUrr1q0ZO3Ys2dnZ1K1bl0mTJtGgQQMmTpxIWloaL774Itdffz21atUiLS2Nbdu28eSTTzJo0KBi/57AQ+AQkd8Dd+HM870UZ5Tcb4HzS5QDY8KlZn24/FXoPAw+uQfeudx5cPCCxyDRhsgv1z4d7YxrVhxvXOx//UmnwoWPB0zWq1cvnnnmGe68807S0tLIysoiJyeH+fPnc845zmSnn3/+OX36OF8+7rjjDoYPH87w4cOZMGECd955J9OmTQt4/I0bN7Jx40ZatWoFQEZGBgsWLCA6Opo+ffowbtw4kpOTWbRoEbfffjtz5swhNTWVSy655Dc3+uzsbApGwtizZw8LFy5ERHjttdd48skneeaZZ44799atW/n6669ZvXo1qampZRc4cILGGcBCVT1PRNoCj5Xo7MaUhVPOhdsWwNf/ch4cXDcb+jwCKTfa0+fmmK5du5Kens7+/fuJj4+nS5cupKWlMX/+/GMTK3322WfccMMNAHz77bd8+OGHAFx77bX88Y9/9Hvc999/n6+//pr4+HheeeUVTjzR6W80ePBgoqOjOXjwIAsWLGDw4MHH0mRlZQXM51VXXXVsOSMjg6uuuoqtW7eSnZ1Nixb+WxYGDhxIVFQU7du3Z/v27R5+K/55CRxHVfWoiCAi8aq6WkTalDgHJvIy0gNvC8ezGJEQEw+9H4BTBzmljxn3wWejId/PwAf21HnkBSkZAMHHR7sh8JDrwcTGxtKiRQsmTpxIz5496dSpE3PnzmX9+vXHxnBavHgxL7/s7YHXgjaOwmrUcCZQzc/Pp06dOkHnAfGXDpxSzz333ENqairz5s1jzJgxftP4Dtle1PiEoQhp6lhXhjsb4DRgtoh8DGwOlkBEBojIGhFZLyLHNe+LyK0i8qOILBWRr0Wkvbu+n4iku9vSReR8nzTz3GMudV+V5M4WIVkH4IOboHZTeGAzjNn321dlu4HWbQnXToMrXvcfNMCeOq/CevXqxdNPP80555xDr169GDduHKeffjoiwooVK2jbtu2xUW179uzJ5MmTAZg0aRK9evUq1jlr1apFixYt+Pe//w04N/Zly5wx4AoPy17Yvn37aNy4MQBvvvlmsc5fHCEHDlX9naruVdUxOLMCvg4MDLS/iETjzE9+IdAeGFoQGHy8q6qnqmpn4EngWXf9TuBSVT0VGA68XSjdMFXt7L7sv7wkPrkP9m6GK16DanUinZuyIeKUPEzFFagkXMIScq9evdi6dSs9evSgQYMGJCQkHAsIn376KQMGDDi27wsvvMAbb7xBp06dePvtt3nuueLPZzdp0iRef/11TjvtNDp06MDHH38MwJAhQ3jqqac4/fTT2bBhw3HpxowZw+DBg+natSv16tUr9vm9KnJY9WM7inTHmVv8gPu+FtBOVRcF2L8HMEZVL3DfPwigqn6nshKRocB1qnphofUC7AIaqmqWiMwD7lPVkMdJ9zqsepXxwxTnyeveDzoDBlY1wao7eoyC04bCSR3LLj9VXHkfVr1fv3689dZbxZoWtiLwMqy6l6qql4GDPu8PuusCaQz84vM+w11XOGMjRWQDTonjTj/HuQL4TlV9W4vecKup/iwBph8UkREikiYiaTt27AiSzSpq90/w33ugWQ/odV+kc1P+LBoH486CcWfDt//ndO01Vdrs2bMrbdDwykvjeMH0sYAzWq6IlHhGHVV9CXhJRK4GHsapmnJOKNIBeALo75NkmKpuEZFE4APgWuAtP8d9FXgVnBJHSfNZqeTlwAe/B4lyuqvaxEjHu3ctLJ8KS9+FmQ/C7D9Dcn+nFNL6AvhnRxvO3VRZXu4YG0XkTv5XyrgdZxbAQLYAvmNUNHHXBTLZ59iISBPgI5zqq2OVe6q6xf15QETeBc7ET+AwQcx73Jn/e/BEZ9DCqirQU+c1kqBGXeh2i/PKXOUEkB+mOHOCVDsBjuzxf0xrWDdVgJfAcSvwPE6pQIEvgBFB9l8CJItIC5yAMQS42ncHEUlW1YKvZxcD69z1dYBPgNGq+o3P/jFAHVXdKSKxwCXA5x6uwfw033mW4fRrnTGoqrJQSwZJ7aD/o9DnL7BxHiydBCs+DGvWqipVJUDtswkjr110vUzklIlz8w91/1wRGQXMBKKBCaq6QkT+BqSp6nRglIj0BXKAPfyvmmoU0Ap4REQecdf1Bw4BM92gEY0TNMaHmqcq7/Bu+HCE0yX1wicinZuKJzoGkvs6r2CBY+qNzjzrTVLgpE4Qm/Db7REcZ6k8S0hIYNeuXdStW9eCRxlSVXbt2kVCQkLRO7u8DDnSGqcqqYGqdhSRTkCqqo4NkqEZwIxC6x7xWb4rQLqxQKDjdg01z8aHKky/Aw7tgKGfQ1yNotOY4vl5oTPJFEBUrNMzqyCQNE4JyzhLlUGTJk3IyMjAOrOUvYSEBJo0aRLy/l6qqsYD9wOvAKjqD24bQ8DAYcqR9Ddg9X+h/9+hUedI56Zyu2cl7N/qtCNlpMGWdFj2HiyxwnEwBU9um/LPS+CorqqLCxUhAzx6a8qVzNXw2Z+g5fnQ/fZI56ZyCNawDlCrIdS61JkpESA/D3asdgLJf/z1Ojem4vASOHaKSEvcecdFZBCwNSy5MqUn56gzpEh8TRg4zpmz25Sc17aIqGho0MF5WeAwFZyXwDES57mItiKyBfgJKHoKLBNZn/8Fti+Hq/8NiQ2K3t9EVuYqpxeXMeWYl7GqNqpqX6A+0BY4Fzg7XBkzpWDtTOcJ6G63Qev+Re9vykbA8ZSiYHwfWPlxmWbHGK+KLHG4Y1KNxBku5GOcLrAjgXuBH4BJ4cygKaYD22DabdDgVOj310jnxvgKVM21/1d4/1qYcp0z1e15D9l8IaZcKnKQQ3f49D04s/31AZIAAe5S1aXhzmBpqDKDHNrzARVfbpYzV8h3b0GrflVr1GJT7gQa5DCUNo5T3OHNEZHXcBrEm6nq0VLOoykpez6g4ouJh0ufh4ad4dMHYPx5MORda/cw5UoobRw5BQuqmgdkWNAwJoxE4Iyb4Pr/QtZBt91jeqRzZcwxoQSO00Rkv/s6AHQqWBaR/eHOoDFVVrPucMuXTmljyrXwxaPO8yDGRFiRgUNVo1W1lvtKVNUYn+VaZZFJY6qsWo3ghhnOoJTzn4b3hsCRvZHOlanibCIGY8q7mHhIfcEZKuaTe+GJk4/fxzpAmDJkjxFXFrnZzsRM/pRwHmZTDojAGb8PvN06QJgyZCWOymL5VNB8uOYDaNU30rkxxlRiVuKoDFRhwQuQ1AFa9ol0bkykbPzS+SwYE2YWOCqD9V9A5kroeYdTpWGqprdS4fX+sG62BRATVhY4KoMFz0FiI+h4RaRzYiLpoqedYUsmDYJXe8Oq/0J+fqRzZSohCxwV3a9L4aevoPutEBMX6dyYcAvU0aFGEpx5M9z5vdMD6+heeH8YjDsbln9oz3+YUmWN4xXdghcgLhG6Xh/pnJiyUFSX25g46HIdnHa1M33t/Kdh6g1Qr7UzcOKsP9t4ZqbEwho4RGQA8BwQDbymqo8X2n4rzki7ecBBYISqrnS3PQjc5G67U1VnhnLMKmXvz7DiI+hxOyTUjnRuTHkSHQOnXQWnDnKGaf/qafjolsD7h9Kd1wbRNK6wBQ4RiQZeAvoBGcASEZleEBhc76rqOHf/VOBZYICItAeGAB2ARsDnItLaTVPUMauOhS87jeHdbo10Tkx5FRUNHS+H9gNh7acw+erA+859DOJqQlwNiE/0Wa7plGptEE3jCmeJ40xgvapuBBCRycBlwLGbvKr6jnVVA3daWne/yaqaBfwkIuvd41HUMauMI3sg/U3oOAhqN4l0bkx5FxUFbS8Ovs+XT5RNXkyFF87A0Rj4xed9BtCt8E4iMhK4B4gDzvdJu7BQ2sbucpHHdI87AhgB0KxZM++5L+/SJkDOIacLrjGl4ZE9zmcq6yBkH4LsAz7LB525642hHDSOq+pLwEsicjXwMDC8lI77Ks4c6aSkpFSuTu25WbDoFWh5PpzUMdK5MZVFVJRTRRWf6H+7BQ7jCmd33C1AU5/3Tdx1gUwGBhaR1usxK6cfpsDB7dDzzkjnxFQ0wbrzGhOicJY4lgDJItIC5+Y+BPhNy5yIJKtqQXeMi4GC5enAuyLyLE7jeDKwGGfK2qDHrPTy850uuCedCqf0jnRuTEVTkt5PNZL8N4THVCv+MU2FFLbAoaq5IjIKmInTdXaCqq4Qkb8Baao6HRglIn1xZhncg1tN5e43BafROxcY6c4+iL9jhusayqX1s2HnGrh8vA0vYsqWv6Dz2YOw8P9g1X+g3aVlnycTEaJVYEyblJQUTUtLi3Q2SscbF8OeTXDXUoiOjXRuTFWXmw0T+sPujXDr11CnEnZEqcJEJF1VUwqvtyFHKpIt6bD5a+h+mwUNUz7ExMGgCU4V6tSbIC8n0jkyZcACR0Wy4AWIrw1dS6XjmTGl48RTIPU5yFgMc/8e6dyYMmCBo6LY/ZMzdETK9YG7SxoTKR2vgC7D4et/OsP8m0ot4s9xmBAtfBkkGrrdFumcGOPfgMfhl8XOmFi3fgOJDUrnuDZGVrljJY6K4PBu+P5t6HQl1GoY6dwY419cdRg80Xna/MPfl95Q7jZGVrljgaMiWPI65ByGHqMinRNjgktqCxc96cwRM//ZSOfGhIkFjvIu5ygsfgVa9YMG7SOdG2OKdvq1cOpgmPcYbF4Q6dyYMLDAUd79MBkO7YCzbHgRU0GIwCX/hBOawwe/d6pai2P7Cpg0uFSzZkqHBY7yLD8fFrwIDU+D5r0inRtjQhefCIPecL70TLsNvDxovG8LTBsJL58FPy8Kvm9uVsnyaYrFAkd5tvYz2LXOGczQhhcxFU2jztDvUedzvPDlovc/shc+HwMvdIEfp0CPkc4ICcEGYJwy3Hl63ZQp645bni14Hmo3c2ZvM6Yi6naL01A++xFo1h0adzl+n9wspwPIV086E5R1ugrOewhOONnZHqjL7eLxMOM+Z071wRNtNIUyZGNVlSfWX91URod3w1MtQfOP3xZfC6rVgb0/wynnQb+/OlWzoVr4Mnw2Gjpc7gz8GW3fhUtToLGq7Ldcnlh/dVMZVT/Rf9AAyNoPdU6Gaz6EVn28H7v7bc74WLP/7JQ4Br7szLNuwsoChzEmsm75ypl9sLjOuhPysmHOoxAVC6kvlOx4pkgWOIwxkVUaN/lz7nNKHl8+7lRXXfKv0ulQYtXHflngMMZUDr1HOyWPr591Sh4XPVXy4GHVx35Z4CgP8vPhizGRzoUxFZsI9HnECR7fvui0eVzwmHVlDwMLHJGWc9R5QGrFh87czblHjt8nWD92YyqCQPOVl/ZnWwT6j4X8XGdK2+hY6PtXb8Ej66DThXjdrNLNWyUS1sAhIgOA53DmB39NVR8vtP0e4Pc484rvAG5U1c0ich7wT59d2wJDVHWaiEwEzgX2uduuV9Wl4byOsDm8GyYPg58XOB/us+6yb0emcirL9gARZ4j3vBz45jmIjofzHwq8vyrs2uAEinWzYPM3TqklrmbZ5bmCCVvgEJFo4CWgH5ABLBGR6aq60me374EUVT0sIrcBTwJXqepcoLN7nBOB9YBv+L9fVaeGK+9lYs8meGcQ7N0MV7wOpw6KdI6MqTxE4KKnnQDw1ZPOq7CEOs7DhutmwZ6fnHX12sCZIyC5PzTrAWPrl2m2K4pwljjOBNar6kYAEZkMXAYcCxxugCiwELjGz3EGAZ+q6uEw5rVsbfkO3r3S+UZ07TRoflakc2RM5RMVBZc+78xl48/RvfDdW9DiHGd4k+R+zsCMvgJVscXVKO3cVijhDByNgV983mcA3YLsfxPwqZ/1Q4DCA/v/XUQeAb4ARqvqcSOdicgIYARAs2bNPGQ7zNZ8ClNvhBr14PoZUL91pHNkTOVVVFffB36C2GqBtxeuYlN1RvxdPhVW/QfaXVryPFZA5eIpGRG5BkgBniq0viFwKjDTZ/WDOG0eZwAnAg/4O6aqvqqqKaqaUr9+OSluLh4Pk6+G+m3g919Y0DAm0oIFDX9E4LIXoXEKfDgCfl0almyVd+EMHFuApj7vm7jrfkNE+gIPAal+Sg5XAh+pak7BClXdqo4s4A2cKrHyLT/fGeRtxn1O3en1n0BN6yllTIUUWw2GvAvV68J7Q2D/1kjnqMyFs6pqCZAsIi1wAsYQ4GrfHUTkdOAVYICq+nuiZihOCcM3TUNV3SoiAgwEloch78UX6ElTgJSb4MInbSA2Yyq6xAYwdDJMuMAJHjd86sy5XkWErcShqrnAKJxqplXAFFVdISJ/E5FUd7engJrAv0VkqYhML0gvIs1xSixfFjr0JBH5EfgRqAeMDdc1FEuwJ0ovfsaChjFlLdCzIiV9huSkjnDFa7B1GUy71alZqCJsWPXSNqZ2kG37Am8zxlRMC16EWQ9Br/ugz58jnZtSZcOqG2NMOPQYCTvXwPynoV5rOO2qkh+znA+uWC56VRljTIUlAhc9A817wfRR8PPCkh+znA+uaIHDGGNKKiYOrnwLajd1hhHasynSOQorCxylLVwNccaY8q36iXD1FGeAxXeHwNH93o+Rlwsb5ha9X4RZG0dpKwf1j8aYCKnXyil5vHO5M0LE0MlF96TMy3FG4105DVb9F47sLpOsloQFDmOMKU2nnOt0vf/PXfBo3eO310iCu1e4weIjWP0JHNnjjMbbegB0GAjv+xu2r/ywwGGMMaWt6/VO4PDnUCY83QqO7oP4WtDmQmh/GbTsA7EJzj6BBldMqBOuHHtigcMYY8pam4ug/UBoeR7ExB+/vXCVd/YhePks0Dxnoqn4yM4VYo3jxhhT1n43DtoM8B80/ImrAQP/D/b+4ox7F2EWOIwxpiI4uSd0vx3SXo94zysLHMYYU1H0+TPUbQUfjyped99SYoHDGGPCIRzPdMVWg4Hj4MCvzvhYEWKN48YYEw7heqar6RnQ8w745jlol+pMeVvGrMRhjDEVTe8/Qf22MP0O5xmQMmaBwxhjKprYBBj4MhzMhM/+VOant8BhjDEVUeMu0OseWPYurPm0TE9tgcMYYyqqc/4IDTo6T6kfLrsxrixwGGNMRRUT51RZHd4Fn/6xzE5rgcMYYyqyhp2ckseP/4aV08vklGENHCIyQETWiMh6ERntZ/s9IrJSRH4QkS9E5GSfbXkistR9TfdZ30JEFrnHfF9E4sJ5DcYYU+71ugdO6gT/vRsO7Qz76cIWOEQkGngJuBBoDwwVkfaFdvseSFHVTsBU4EmfbUdUtbP7SvVZ/wTwT1VtBewBbgrXNRhjTIUQHeuMf3V0H3xyb9hPF84Sx5nAelXdqKrZwGTgMt8dVHWuqh523y4EmgQ7oIgIcD5OkAF4ExhYmpk2xpgKqUEHOO9BZ0Ko5R+G9VThDByNgV983me46wK5CfDtU5YgImkislBEBrrr6gJ7VTW3qGOKyAg3fdqOHTuKdQHGGFOh9LwLGnVxSh0H/cznUUrKxZAjInINkAKc67P6ZFXdIiKnAHNE5EdgX6jHVNVXgVcBUlJStDTza4wx5VJ0DOzZ5Ew/+3Tyb7fVSCq1YVDCWeLYAjT1ed/EXfcbItIXeAhIVdWsgvWqusX9uRGYB5wO7ALqiEhBwPN7TGOMqbICzVnub0bBYgpn4FgCJLu9oOKAIcBv+oqJyOnAKzhBI9Nn/QkiEu8u1wPOAlaqqgJzgUHursOBj8N4DcYYYwoJW+Bw2yFGATOBVcAUVV0hIn8TkYJeUk8BNYF/F+p22w5IE5FlOIHicVVd6W57ALhHRNbjtHm8Hq5rMMYYc7ywtnGo6gxgRqF1j/gs9w2QbgFwaoBtG3F6bBljjIkAe3LcGGOMJxY4jDGmMgnHzIOFlIvuuMYYY0pJuGYe9GElDmOMMZ5Y4DDGGOOJBQ5jjDGeWOAwxhjjiQUOY4wxnogzikflJiI7gM3FTF4PKO7MKJFKG8lz2zVXjLSRPLddc8VIC85gs/WPW6uq9gryAtIqWtqKmm+7Zvt92TWXn7TBXlZVZYwxxhMLHMYYYzyxwFG0Vytg2kie2665YqSN5LntmitG2oCqROO4McaY0mMlDmOMMZ5Y4DDGGOOJBY4gRGSAiKwRkfUiMtpDugkikikiy4txzqYiMldEVorIChG5y0PaBBFZLCLL3LR/Lcb5o0XkexH5bzHSbhKRH93ZHNM8pq0jIlNFZLWIrBKRHiGma+Oer+C1X0T+4OG8d7u/q+Ui8p6IJHhIe5ebbkUo5/T3uRCRE0Vktoisc3+e4CHtYPfc+SKS4vG8T7m/6x9E5CMRqeMx/aNu2qUiMktEGoWa1mfbvSKi7vTQoZ53jIhs8fl7X+TlvCJyh3vdK0TkSY/X/L7PeTeJyFIPaTuLyMKC/w0R8TsZXYC0p4nIt+7/1n9EpFaAtH7vHaF+xjwJRx/fyvACooENwClAHLAMaB9i2nOALsDyYpy3IdDFXU4E1no4rwA13eVYYBHQ3eP57wHeBf5bjLxvAuoV8/f9JvB7dzkOqFPMv9k2nIeWQtm/MfATUM19PwW4PsS0HYHlQHWc6Qk+B1p5/VwATwKj3eXRwBMe0rYD2gDzgBSP5+0PxLjLTwQ6b5D0tXyW7wTGhZrWXd8UZ1rpzYE+MwHOOwa4L4S/j7+057l/p3j3fZKX9IW2PwM84uHcs4AL3eWLgHke0i4BznWXbwQeDZDW770j1M+Yl5eVOAI7E1ivqhtVNRuYDFwWSkJV/QrYXZyTqupWVf3OXT6AM1974xDTqqoedN/Guq+Qez+ISBPgYuA1T5kuIRGpjfMP8zqAqmar6t5iHKoPsEFVvYwSEANUE5EYnCDwa4jp2gGLVPWwquYCXwKXB0sQ4HNxGU7QxP05MNS0qrpKVdcUldEAaWe5+QZYCDTxmH6/z9saBPicBflf+Cfwx0DpikhbpABpbwMeV9Usd5/M4pxbRAS4EnjPQ1oFCkoKtQnwOQuQtjXwlbs8G7giQNpA946QPmNeWOAIrDHwi8/7DEK8gZcWEWkOnI5Tcgg1TbRbhM4EZqtqyGmBf+H8M+d7SONLgVkiki4iIzykawHsAN5wq8leE5EaxTj/EAL8M/ujqluAp4Gfga3APlWdFWLy5UAvEakrItVxvkU29ZhfgAaqutVd3gY0KMYxSupG4FOviUTk7yLyCzAMeMRDusuALaq6zOs5XaPcarIJHqtdWuP8zRaJyJcickYxz98L2K6qXmZM+gPwlPv7ehp40EPaFfzvS+tgQvicFbp3lPpnzAJHOSUiNYEPgD8U+nYXlKrmqWpnnG+QZ4pIxxDPdwmQqarpxcmv62xV7QJcCIwUkXNCTBeDUzx/WVVPBw7hFKlDJiJxQCrwbw9pTsD5h2wBNAJqiMg1oaRV1VU4VTyzgM+ApUCelzz7OabioYRYGkTkISAXmOQ1rao+pKpN3bSjQjxfdeBPeAg0hbwMtAQ64wT7ZzykjQFOBLoD9wNT3NKDV0Px8AXFdRtwt/v7uhu3dB2iG4HbRSQdpwoqO9jOwe4dpfUZs8AR2BZ+G9mbuOvCTkRicf7wk1T1w+Icw63qmQsMCDHJWUCqiGzCqZY7X0Te8XjOLe7PTOAjnOq+UGQAGT6lo6k4gcSLC4HvVHW7hzR9gZ9UdYeq5gAfAj1DTayqr6tqV1U9B9iDU6fs1XYRaQjg/gxYfVLaROR64BJgmHtDKa5JBKg+8aMlTqBe5n7WmgDfichJoSRW1e3ul6N8YDyhf8bA+Zx96FbpLsYpWfttmA/ErdK8HHjfSzpgOM7nC5wvNyHnW1VXq2p/Ve2KE7A2BMmfv3tHqX/GLHAEtgRIFpEW7rfZIcD0cJ/U/Qb0OrBKVZ/1mLZ+Qe8YEakG9ANWh5JWVR9U1Saq2hznWueoakjfvt3z1RCRxIJlnMbXkHqVqeo24BcRaeOu6gOsDPXcruJ8C/wZ6C4i1d3fex+ceuGQiEiS+7MZzs3kXY/nB+czNdxdHg58XIxjeCYiA3CqJVNV9XAx0if7vL2M0D9nP6pqkqo2dz9rGTgNuttCPG9Dn7e/I8TPmGsaTgM5ItIapxOG15Fj+wKrVTXDY7pfgXPd5fOBkKu5fD5nUcDDwLgA+wW6d5T+Z6ykreuV+YVTb70WJ8I/5CHdezjF6Bycf4ybPKQ9G6co+QNO9cdS4KIQ03YCvnfTLidAr48QjtMbj72qcHqfLXNfK7z8vtz0nYE0N+/TgBM8pK0B7AJqF+Na/4pz01sOvI3b4ybEtPNxAtwyoE9xPhdAXeALnBvJ58CJHtL+zl3OArYDMz2kXY/ThlfwGfPbKypI+g/c39kPwH+AxsX5XyBIT7wA530b+NE973SgoYe0ccA7br6/A873cs3u+onArcX4O58NpLuflUVAVw9p78K5D60FHscd8cNPWr/3jlA/Y15eNuSIMcYYT6yqyhhjjCcWOIwxxnhigcMYY4wnFjiMMcZ4YoHDGGOMJxY4jPFIRPLktyPyjnbXzxNnNOVlIvJNwXMpIhInIv8SZ5TldSLysTsuWMHxThKRySKywR2uZYaItBaR5nL86K5jROQ+d7m7O3zGUnFGFB5Thr8GU4XFRDoDxlRAR9QZ1sWfYaqa5o7V9RTOMCiP4QwV0UZV80TkBuBDEenmpvkIeFNVh4AzjDbOeEK/HH/433gTuFJVl4lINM5IucaEnQUOY8LjK+AP7thMNwAtVDUPQFXfEJEbcZ4gViBHVY89Dazu4H/uQHXBJOE8LIZ7bK9P2xtTLBY4jPGumvx2Ep9/qGrhsYsuxXnCuRXwsx4/UGUa0MFdDjawZMtC5zoJZ3RVcIYmXyMi83AGWnxTVY+GehHGFJcFDmO8C1ZVNUlEjuAMpXEHUNLZ1jb4nsu3HUNV/yYik3DGBbsaZ7yu3iU8nzFFssBhTOkapqrHps0Vkd1AMxFJVGdynQJdgYLpeQcV92SqugF4WUTGAztEpK6q7iru8YwJhfWqMiaMVPUQTiP2s24DNiJyHc5sg3PcV7zvxFci0klEehV1bBG52Gc+iWSc+UD2lu4VGHM8CxzGeFetUHfcx4vY/0HgKLBWRNbhzOL2O3XhjHLb1+2OuwL4B85MbUW5FqeNYynOqLHDChrgjQknGx3XGGOMJ1biMMYY44kFDmOMMZ5Y4DDGGOOJBQ5jjDGeWOAwxhjjiQUOY4wxnljgMMYY48n/A69wHWo+JstmAAAAAElFTkSuQmCC\\n\",\n      \"text/plain\": [\n       \"<Figure size 432x288 with 1 Axes>\"\n      ]\n     },\n     \"metadata\": {\n      \"needs_background\": \"light\"\n     },\n     \"output_type\": \"display_data\"\n    },\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 229 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"value1 = ft_recall_10\\n\",\n    \"value2 = cs_recall_10\\n\",\n    \"name = \\\"Recall@10\\\"\\n\",\n    \"\\n\",\n    \"length = len(value1)\\n\",\n    \"plt.plot(range(length), value1, \\\"-s\\\", label=\\\"Finetune\\\")\\n\",\n    \"plt.plot(range(1, length), value2, \\\"-s\\\", label=\\\"w/o Pretrain\\\")\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"plt.xticks(range(length))\\n\",\n    \"plt.ylabel(name)\\n\",\n    \"plt.xlabel(\\\"EPOCHS\\\")\\n\",\n    \"plt.legend()\\n\",\n    \"plt.show()\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 154,\n   \"id\": \"attached-printer\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"image/png\": \"iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA1E0lEQVR4nO3deXgUVfbw8e/JHiBhC2HXALIICgEiIAoyIMiog7soqCDOuAsuPxVHR3F5x93RGR13XFFQVMAVccENEQKiEkAFRA3DEhANCNnP+0dVoA3dne5OL1nO53n6SXdV3aqTTqdO17237hVVxRhjjKkqLtYBGGOMqZ0sQRhjjPHKEoQxxhivLEEYY4zxyhKEMcYYrxJiHUC4ZGRkaFZWVqzDMMaYOmXZsmXbVLWVt3X1JkFkZWWRm5sb6zCMMaZOEZEffa2zKiZjjDFeWYIwxhjjlSUIY4wxXlmCMMYY45UlCGOMMV7Vm15MxtQ2ObctYNuukv2WZzRJIveGkfXuuKb+sQRhjB81Odl6K+dveW04rjGeLEGYWi+W34iDOdmWllews6iMwj2lFBaV+t3v5+u2k5IYR0piPMkJzk/P5/6O++vuEvaUlrO7pJw9JeUUVT4vdV7Hkl011S+WIEytF+1v4hUVyi+7SyjYWex33yf/9zMKi8rYWVRK4Z4y9pQGfnI+8/HFAW9bVfYtC0Iu+9dncjmkfTqHtm/KIe2b0jo9Zb9t6uLVS129aqrtic0ShKnVqpvQ6v9e/oomyQmkpSTQJDmBxh7PmyQn+D1xPPv5BrYWFlOws5iCXcVs3VlEwc5itu0qobyi+om0GiUl0Do9hfSURNJTE0hPSSQtJYH01ETSUxL567O+7+x/4W8DKS6toLisnKLSCopKnSuB4rIKikor+Nd73/kse+PxPWmUFE9qUjypic7PRknOFUhqYjzD7/3IZ9kftu3i/TVbqHxbW6Ulc0g7J2H0at+UQ9s3rfZku7ukjO27SijYVcy2ncVs/71k709/PvqugA7NU2nfLJWUxPj91gd7slRVft1dypadRX6PW5vFqhoyUJYgTK3z6+4SPlu7nU++L+CT77f53XbR2m3sLC5jV3EZwU6OeOPcPOIEMpok0yrNefRsm+48b5JMZnoKF89Y7rP8838dGNwBPQzukuF3vb8EMenITiEf9/2rhvF7cRmrNhWycuNvrNzo/PzouwICyIn0vPEddvuoxkpL8X86mTB9yd7nGU2SaN+8ER2apTpJo3mq35PlIx+tY0thEVsLi9lSWMSWnUVsKSympKyi2phPePBTurdJo3ubdA5uk0b3Nmm0bJK8d31NT7Shlq/uy8/vxWU0SopHRLyuj8ZVkyUIExX+/okWTR3B8p928Mn3BXz6/Ta+3vgbqs4JZ3CXlmz8dY/P/S66bgTg/LPtLinn9+IyJ2EUOUlj/BNf+Cy79PqjadE4ifg47/+ANZXRJMnn7xxJ1R23cXICh2W14LCsFnvX7SkpZ/XmQvI2/sY/5ub53PeZAw4go0kyLZsk0cr9Wfk6OSGerKlv+iz70gWHs/HX3WzcsYf8HXvY+OseVm8qZMHqLdWe6O94ew1NkhPITE+mdVoK/Q9oTuv0FDLTU2iTnsIlL/hO5I2TE3h/9VZeys33eC+S6dEmjR5t0mp8ovVXftG6bU5CcxNb1QTnT6+b5hMnTvxpyQk0qXKVHA2WIExU+Psnyr7lXXaXlBMfJ/Tt2IwpI7oypGsr+nRoSkJ8nN+TTiURobH7z5MZYEyt0pKr3aYmJ/maXOZH+7ipSfH0O6A5/Q5o7jdB/OP4nkHvu9KATi2AFvstr6hQtv1ezID/977PsitvPoYmyb5PV5e84Pu4L/xtEAAFO4v5dvNO1mwuZM3mnXy7eSfPLfY5Th0Ao/71EXEiiAgCxMXxx9fVfLcY9/i+LyiNk+Jp3TTlDwnu0Y/X+yz792N7sKvoj194dhWXsbOojE2/RadazRKEiajS8gp+3P67321O6deBI7tmcHiXlqSnJO63PlbfxKFmJ/m6eNyaCuVvFRcnZKbt31juyV9yCPS4ldWIR3bdV71XXqF0+ftbPvfbOaMJilKhzlWq58+KAOo0X/jbQFqnp9A6PcXr7+AvQZw/tIvffQfyxammLEGYgPmrJvr02uGsL/idtQW7WLtlJ99v3cX3W3exYdvvlFVTuX3riYf4XR+rb+INVayummoi1ONWV734yNn9q92HvxN1dW1Ntf3zaQnCBMxfNVHPG9/Z28gZJ3Bgy8YclNmEkT1b0zWzCVe+9FUUI92nrn4Tj6VYvWe1/WQZCbX9y48lCBOQ6hoRLx3ela6ZTTgoswmdMhrv140xVgnC1B11NTHFKrFF4/2yBGH82lJYxIwvfuKFL37yu92VI7v5Xd8Qvx2auqGmJ9r6fJVqCcLsR1VZumEHz3y+gfkrN1OuyrBurfjw24KQ91mf/4mMqa8sQZi9dpeUMXfF/3hm0QbWbN5JekoC5x6RxVmDDuTAlo2j0mvCGFN7WIJoYHz1REpNjCMxPo7CojIObpvOHScfygnZ7UlN2teWYNVExjQsEU0QIjIaeACIB55Q1Tu8bHM6MA1Q4CtVHecunwDc4G52m6o+E8lYGwpfPZH2lFYw4uDWTBicRc6Bzb3e3m/VRMY0LBFLECISDzwEjATygaUiMk9VV3ls0xW4DjhCVXeISKa7vAVwE5CDkziWuWV3RCrehmDbLv+39j84rl+UIjHG1AWRvIIYAKxV1fUAIjITOAFY5bHN34CHKk/8qrrVXX4MsEBVf3HLLgBGAy9GMN56Z/uuYpb88Aufr9/O4vXb+W7LrliHZIypQyKZINoDP3u8zgeqDn/ZDUBEPsOphpqmqu/4KNu+6gFE5HzgfIADDjggbIHXZv7uZn73iqNY8sN2Fq//hc/XbefbLTsBaJQUT05WC07s25673vk22iEbY+qoWDdSJwBdgWFAB+BjETk00MKq+hjwGEBOTk6Qgz3XTf7uZu53qzORTGpiPDlZzRmT3Y5BnVvSu0NTEuPjACxBGGMCFskEsRHo6PG6g7vMUz7whaqWAj+IyHc4CWMjTtLwLLswYpHWE/83qhuHd2nJoe2bkZQQ53Ub64lkjAlUJBPEUqCriHTCOeGfAYyrss0c4EzgKRHJwKlyWg+sA/4pIs3d7UbhNGYbPy4d3rXabawnkjEmUBFLEKpaJiKXAvNx2hemq2qeiNwC5KrqPHfdKBFZBZQDV6vqdgARuRUnyQDcUtlg3ZD9urt2z69rjKlfItoGoapvAW9VWXajx3MFrnQfVctOB6ZHMr665Kftu5n49JLqNzTGmDDxXlFtapUVP//KyQ9/xvZdJTRN3X9CHbA2BGNM+MW6F5Opxvy8zUyZ+SWZaSk8de5hdGnVJNYhGWMaCEsQtdj0T3/g1jdX0adDM56YkENGk+rnUDbGmHCxBFELlVcot725iqc+28AxvVpz/9i+fxg0zxhjosESRC2zp6ScKTO/5N1VW5h0RCeuP+7gaufNNcaYSLAEUYts21XMec/k8nX+r9z0l56ce0SnWIdkjGnALEHUEusKdjHxqSUU7Czm0bP6M6pXm1iHZIxp4CxBxICvAfcEeO2SI8ju2CzqMRljTFV2H0QM+BpwT8GSgzGm1rAEYYwxxitLEMYYY7yyBGGMMcYrSxBR9uVPNq22MaZusAQRResKdjHp6aX4uu/NBtwzxtQm1s01SrYUFnHOk0uIE+GDq4aRldE41iEZY4xfliCioLColAnTl7Bjdwkzzx9kycEYUydYFVOEFZWW87dnclm7dRePnNWf3h2axTokY4wJiF1BRFB5hXLFrBV88cMv3D82m6HdWsU6JGOMCZhdQUSIqjJtXh5vr9zMDccdzIl928c6JGOMCYoliAj5zwdreW7xj1wwtDN/HdI51uEYY0zQLEFEwItLfuK+Bd9xct/2XDu6R6zDMcaYkFiCCLN38zZz/WvfcFS3Vtx5am/ibLIfY0wdZQkijHI3/MJlL37Joe2b8t/x/UiMt7fXGFN32RksTL7bspNJTy+lXbNUpk88jMbJ1kHMGFO32VksRL4m/UmIE1o2SY5BRMYYE152BREiX5P+/LK7NMqRGGNMZFiCMMYY45UlCGOMMV5ZgjDGGOOVJYgQbNj2e6xDMMaYiLMEEaSi0nIunrEcX7e/2aQ/xpj6wrq5Bunm1/NYtamQ6RNzGN6jdazDMcaYiLEriCC8siyfF5f8zMXDulhyMMbUe5YgArRmcyHXz/mGgZ1acOXIbrEOxxhjIs4SRAB2FZdx8YzlpKUk8p9xfUmwMZaMMQ2AnemqoapMfeVrNmz7nX+f0ZfMtJRYh2SMMVFhCaIazy3+kTe+3sRVo7pzeJeWsQ7HGGOiJqIJQkRGi8i3IrJWRKZ6WT9RRApEZIX7+KvHunKP5fMiGacvK37+lVvfWMXwHplcdFSXWIRgjDExE7FuriISDzwEjATygaUiMk9VV1XZdJaqXuplF3tUNTtS8VVnx+8lXDJjOZlpKdx3eh+b+McY0+BE8gpiALBWVderagkwEzghgscLm4oK5cqXVrB1ZxH/Hd+PZo3s5jdjTMMTyQTRHvjZ43W+u6yqU0TkaxGZLSIdPZaniEiuiCwWkRO9HUBEzne3yS0oKAhb4A9/tI4Pvy3gH8f3pE/HZmHbrzHG1CWxbqR+HchS1d7AAuAZj3UHqmoOMA64X0T2awRQ1cdUNUdVc1q1ahWWgD5ft5173/2Wv/Rpx9mDDgzLPo0xpi6KZILYCHheEXRwl+2lqttVtdh9+QTQ32PdRvfnemAh0DeCsQKwtbCIy178kqyMxtx+8qGIWLuDMabhimSCWAp0FZFOIpIEnAH8oTeSiLT1eDkGWO0uby4iye7zDOAIoGrjdliVlVdw2Ytf8ntxGY+c1Z8mNqe0MaaBi9hZUFXLRORSYD4QD0xX1TwRuQXIVdV5wGQRGQOUAb8AE93iBwOPikgFThK7w0vvpxrzNa/0uMcXk3vDyHAfzhhj6pSIfk1W1beAt6osu9Hj+XXAdV7KLQIOjWRs4HteaV/LjTGmIYl1I7UxxphayhKEMcYYr6wl1kTH3V3h9637L2+cCVd/H/14jDHVsisIEx3ekoO/5caYmGvQCcLX/NE2r3QYlZXA/76MdRTGmBA06Com68oapOqqicrLoGCNkxD+t9z5uSUPyq1XmDF1UYNOECZI/qqJnhgJm7+Bsj3OsuR0aNsHBl0E7frCyxN977esGBKSwx6uMaZmLEGY8IhLgJxJTjJo1xdadIY4jxpMfwni8RFwyhOQ2SPiYRpjAhdQghCRY4AT2Tca60Zgrqq+E6G4TG1SXgorX/G/zaS3/a9vnOn9CiSlKez8Hzx2FBzz/yDnPLAxsIypFapNECJyP9ANeBZnyG5wBt6bLCJ/VtUpkQvPxFTxLlj+LHz+EBTmV7+9P/66su7cAnMugjevgu/fgxMehMYZNTueMabGAunFdKyqHquqM1X1U/cxEzgOODbC8ZlY2FUAH9wG/+oF86+D5gfCuJcid7y01jB+NhxzO6x7Hx4eDGvfj9zxjDEBCaSKqUhEDlPVpVWWHwYURSAmEyvb18HnD8KKF5yG4x7HwRGXQ8fDnPW+qokaZ9b82HFxcPjF0GkovHIePH8yDLoEjr7JGrCNiZFAEsRE4GERSWNfFVNH4Df2jb5q6gpfXVXjk6Gi1Gls7nMmDL4MMrr+cZto3PHc5hA4fyEsuBEWPwQ/fASnPGkN2MbEQLUJQlWXAwNFpA0ejdSqujmikZnI8NVVtbwYjrwCBl4IaW2iG1NVialw7N1w0NEw52L470Dv29kwHcZEVEB3UrvJAVVdBvwEDBaRnpEMzMTA0dNinxw8dTsGLlrke70N02FMRAXSi+kCYKrzVO7EqVZaCdwuInep6pORDdE0aGmtYx2BCaPS0lLy8/MpKrLmy2hLSUmhQ4cOJCYmBlwmkDaIS4FeQCrwI3CQqm4WkebAh4AlCBM7W/Kgda9YR2EClJ+fT1paGllZWTbnexSpKtu3byc/P59OnToFXC6QKqZSVd2tqtuBdZVtD6q6A9DQwjUx8W09vK/x4cHw7Imw9j1Q+zjWdkVFRbRs2dKSQ5SJCC1btgz6yi2QBKEiUnlNcpzHAVMCLG9qg19/gtcucHopeROOrqqxMOJG2Loanj/FSRZfPu900TW1liWH2AjlfQ+kiukk3CsFVfW8nbYlcFXQRzTRV1YCL58LFeVwyRJo2SXWEQXH3/0XQ66Cwy+DlbNh0YMw9xJ4/xYY8Ddn2I5GLaIfr6nV4uPjOfTQfVPez5kzh3HjxrFokZ8OEX48/fTTjBo1inbt2oUrxFojkG6uP3m+FpGWwA5V3YgzJpOp7d6bBhtz4bRn6l5ygOq7siYkQfY45/6N9Qudm/0+uA0+vtdZXznCrCfrIlvr5dy2gG279h8qPqNJUo2G6k9NTWXFihV/WBZqcgAnQRxyyCH1MkEE2s21uYg8KCIfAQ8Bb4vIdBFpHNnwTI2tfsO54WzA+dDrxFhHE1ki0OVPcNYrcPFiOPQU78kBrItsHeAtOfhbXhNNmjQBYOHChQwbNoxTTz2VHj16MH78eNRt21q2bBlHHXUU/fv355hjjmHTpk3Mnj2b3Nxcxo8fT3Z2Nnv27CErK4tt27YBkJuby7BhwwCYNm0akyZNYtiwYXTu3Jl///vfe4///PPPM2DAALKzs7ngggsoLy8P++8YikC6uTYD3gL+rqqXeiz/E3CHiLwE5KnqLxGL0oTmlx+cG83a9YVRt8U6mujKPBhOeMhpkzC10s2v57Hqf4UhlR376Odel/dsl85Nf/Hfq23Pnj1kZ2cD0KlTJ1577bU/rP/yyy/Jy8ujXbt2HHHEEXz22WcMHDiQyy67jLlz59KqVStmzZrF9ddfz/Tp03nwwQe55557yMnJqTbuNWvW8OGHH7Jz5066d+/ORRddxNq1a5k1axafffYZiYmJXHzxxcyYMYNzzjknsDcjggJpg/gHcI+qfigizwGDgG1ABvANIMANwJURi9IEr6x43xwMpz1t4xkZ4/JWxeRpwIABdOjQAYDs7Gw2bNhAs2bNWLlyJSNHOlVb5eXltG3bNuhjH3fccSQnJ5OcnExmZiZbtmzh/fffZ9myZRx2mDPm2Z49e8jMrB2dRgJJEENVtbIxuhg4U1VzRaQfcBHwKfBApAI0IXr3Bti0AsbOgOZZsY7GmP1U900/a+qbPtfNuuDwcIezV3Lyvi9T8fHxlJWVoar06tWLzz/3fuXiKSEhgYqKCoD9upX62veECRO4/fbbw/QbhE8gbRApsq9/VD/gK/f5SqCfqlZEJDITurzXYMljzmioBx8f62hqr/LSWEdg6oju3btTUFCwN0GUlpaSl5cHQFpaGjt37ty7bVZWFsuWLQPglVeqmWgLGDFiBLNnz2brVqdd7JdffuHHH38M968QkkASxBJghPv8v8C7IvJPYD7wqIgcBuRFKD4TrO3rYO5l0D7HGVupofN3f8f8v0cvDhO0jCZJQS2PpKSkJGbPns21115Lnz59yM7O3tvzaeLEiVx44YV7G6lvuukmpkyZQk5ODvHx8dXuu2fPntx2222MGjWK3r17M3LkSDZt2hTpXykgotXcfSoinYGXgONUdYuIZACdgfU4CWYeMEFVv410sP7k5ORobm5uLEOIvdIiePJo+PVnuPATaHZArCOqveZf73SHHfMg9Ds71tE0GKtXr+bggw+OdRgNlrf3X0SWqarXFvZA7oNYLyKXAPNE5F1gMVCOM5vcscBVsU4OxjX/Otj8DZw5y5JDdY6+2RnH6c0roVWPfZMiGWP2Cug+CFX9Ajgc+Bg4GDgEJ1EMVtVPIheeCdg3syF3OgyeDN1Hxzqa2i8+AU6dDuntYNZZUFg7LumNqU0C6cUEgNsYvcB9mNpk2/fw+hToONAZm8gEplELOONFeOJoJ0lMfBMSU8K3f1+z99ld3KaOqPYKQkR2ikihx89Cz9fRCNL4UbrHud8hPglOfQriAx/r3QCte8JJjzhDkbx5VXhHhPV1t7bdxW3qiEDaINKiEYgJkK9vpSnNoGn7/Zeb6vUcA0OvgY/vgrZ9YOD5sY7ImFohkKE2/A6HaUNsRJmvb59Fv0Y1jHpnmNvA/85UZ5iOTkNqtr/ysvDEZUwMBdJIvQzIdX9WfTTwfqWm3oiLg5Mfc0a7fXmCM39GKFRh1Tz476Dwxmei6o477mDGjBkBbfv000/TqlUrsrOz6dmzJ48//nhQx5ozZw6rVq0KOsZ58+Zxxx13BF0uGIFUMQU+P50xdVlKutNo/fhwmDkOJr0LSY0CL7/hU1hwk9OekdE9cnE2FDFs5J8/fz4vvfRSwNuPHTuWBx98kK1bt9KrVy/GjBlD69b75lMvKysjIcH76XbOnDkcf/zx9OzZc791/sqNGTOGMWPGBBxjKIKaEc4d9nuAiAytfFSz/WgR+VZE1orIVC/rJ4pIgYiscB9/9Vg3QUS+dx8TgonTmJBlHASnPAGbVzqTDwXSaL15Jcw4DZ4+Dgr/B2P+Axct8n0Xd3ySM3mT8S8Cjfx333333mG2r7jiCoYPHw7ABx98wPjx4wEoLCykpKSEVq1asWHDBoYPH07v3r0ZMWIEP/3k/8oyMzOTLl268OOPP+69w3rgwIFcc801rFu3jtGjR9O/f3+GDBnCmjVrWLRoEfPmzePqq68mOzubdevWMWzYMC6//HJycnJ44IEHeP311xk4cCB9+/bl6KOPZsuWLYBz5XLppc4A2xMnTmTy5MkMHjyYzp07M3v27JDfI08Bd3N1T95TgA7ACpxRXT8HhvvYPh5n7oiRQD6wVETmqWrVa6lZnsOIu2VbADcBOTiz2S1zy+4INF5jQtZtlNNd+P2boW1vOPIK79vt+BE+/Cd8Pcu5+hh5izPvRmKqs97bt9wvHoO3r3bu5P5zZKsHar23pzrtPqF46jjvy9sc6vd9HTJkCPfeey+TJ08mNzeX4uJiSktL+eSTTxg61Pm++9577zFihDO60GWXXcaECROYMGEC06dPZ/LkycyZM8fn/tevX8/69es56KCDAMjPz2fRokXEx8czYsQIHnnkEbp27coXX3zBxRdfzAcffMCYMWM4/vjjOfXUU/fup6SkhMqRIXbs2MHixYsREZ544gnuuusu7r333v2OvWnTJj799FPWrFnDmDFj/rC/UAWcIHCSw2HAYlX9k4j0AP7pZ/sBwFpVXQ8gIjOBE4BAKtuOARZUNoCLyAJgNPBiEPHWPxUVEJcIFV4Gmaurc0rXVkde4Zy83pvmPKpKTHWuAiQOjpgCR14Oqc2r3+/A82HHD7D4v84ou4MuDG/cxq/+/fuzbNkyCgsLSU5Opl+/fuTm5vLJJ5/svbJ45513OPfccwH4/PPPefXVVwE4++yzueaaa7zud9asWXz66ackJyfz6KOP0qKF07fntNNOIz4+nl27drFo0SJOO+20vWWKi33PnT527Ni9z/Pz8xk7diybNm2ipKSETp281/qfeOKJxMXF0bNnz71XGTUVTIIoUtUiEUFEklV1jYj4q2htD/zs8TofGOhlu1PcqqrvgCtU9WcfZa0P59InnORw4sPOFJsmckTghAch71Xv60v3QL9z4KipwXcvHnWb0wj+zlRnSJQex9Y83rqouiuoaU19rzvX91Dg/iQmJtKpUyeefvppBg8eTO/evfnwww9Zu3bt3jGKlixZwsMPPxzUfivbIKpq3NiZdLOiooJmzZr5nYfCWzlwrmKuvPJKxowZw8KFC5k2bZrXMp5DiVc3xl6ggmmDyHdnl5sDLBCRuUBNx6R9HchS1d44d2g/E0xhETlfRHJFJLegoKCGodRyOzY432QPOtqZe9lEXlI1M+qO+U9o957ExcPJjzsz/b1yHmxcHlp8JiRDhgzhnnvuYejQoQwZMoRHHnmEvn37IiLk5eXRo0ePvaOwDh48mJkzZwIwY8YMhgwJrftzeno6nTp14uWXXwacE/hXXzkzJ1QdLryq3377jfbtnc/ZM88EdYqssYAThKqepKq/quo0nFnmngRO9FNkI9DR43UHd5nnPrerauV11hNA/0DLuuUfU9UcVc1p1apVoL9K3aMK8yY71RnH3+98uzV1W1IjGDcLGmfAC2ND71Zbn/mqNq1hdeqQIUPYtGkThx9+OK1btyYlJWXvif/tt99m9Oh9Y5n95z//4amnnqJ3794899xzPPBA6HOjzZgxgyeffJI+ffrQq1cv5s6dC8AZZ5zB3XffTd++fVm3bt1+5aZNm8Zpp51G//79ycjICPn4oah2uO+9G4oMwpl7eqf7Oh042B3Iz9v2CTjVRiNwTu5LgXGqmuexTVtV3eQ+Pwm4VlUHuY3Uy3AmKAJYDvT3d1NevR7ue9nTzlhLx/8LcibFOpqGxV81x7Tfar7/rWvgyVGQ3hYmzYfUZjXfZy1W24f7HjlyJM8++2xI04nWBcEO9x1MFdPDwC6P17vcZV6pahlwKc7EQquBl1Q1T0RuEZHKzruTRSRPRL4CJgMT3bK/ALfiJJWlwC0N9o7t3/Jh/g2QNQT6TYx1NCbcMnvA2OeciZ5eOhvKSmIdUYO2YMGCepscQhFMI7Wox+WGqla4Vwk+qepbwFtVlt3o8fw64DofZacD04OIr/5RhTeuAC136rvjgrptxYRD40zfN2uFS+ejYMy/Yc5F8MblcMJDVo1oaoVgEsR6EZnMvquGi3FmlTOR8vUs+P5dGH0ntLAb2mMiWsNyZ49z7qv46A5o3gmOujo6xzXGj2C+kl4IDMZpT6jssmrDXkbKzi3w9rXQcZBz85Wp/4ZNhd5nwIe3wdeBD/NQ14SrC6YJTijvezATBm0Fzgj6CCZ4qs5UmKV7nL74VrXUMIg4VYmFG51hPtLbQdaRsY4qrFJSUti+fTstW7ZErBotalSV7du3k5IS3IRYwQy10Q2neqm1qh4iIr2BMap6W3ChmmrlvQZr3nDmTc7oGutoTDQlJDmN1nd1ccZ2qqqOz0bXoUMH8vPzqff3LdVCKSkpdOjQIagywbRBPA5cDTwKoKpfi8gLgCWIcPp9G7x1NbTrB4dfWv32pv5Jbe50TPCmjs9GV3kns6kbgqm7aKSqS6oss1lRwu3ta6HoN6cnS3ww+dsYY8IrmASxTUS64IyuioicCmyKSFQN1Zo3YeVsOOoaZ65kY4yJoWC+ol4CPAb0EJGNwA/A+IhE1RDt2QFvXAmtD/U9vLQxxkRRML2Y1gNHi0hjnCuP3Ti9mmo6YJ8BZ36A3wuc8XniE2MdjanNSnYHN9OdMSGqtopJRNJF5DoReVBERuIkhgnAWuD0SAfYIHz/HqyY4cwp0C471tGY2sDfndozz3S6QBsTYdUO1ucO670DZ/a4EUAmIMAUVV0R6QADVacG6/M5124ruHpt9OMxdceKF2DOxXDQCBg7AxKD69duTFX+BusLpIqps6oe6u7oCZyG6QNUtSiMMTYsPufatb7hphrZ46CiDOZdBi+d49wzkZBcfTljQhBIL6a981uqajmQb8nBmBjqdw4cdx98Px9ePhfKvUxBa0wYBHIF0UdECt3nAqS6rwVQVU2PWHTGGO8OO8+ZE/vtq2H2JDh1unVuMGFXbYJQ1fhoBGKMCdLA8505yuf/HV4935nG1G6uNGFkn6ZoKyqsfhtjAnX4JU4V03s3QVwCnPSIM+e1MWFgCSKayophlp97C8M5CY1pOI683LmS+OA2p5ppjI0AbMLDEkS0VFTAaxfADx/DSY9Bn7GxjsjUJ0OvhvIyZ8KhuHg4/gFLEqbGLEFEgyq8c60zjPfIWy05mMgYNtXpAvvJPbD82f3X1/Ghwk302VeMaPjkXljymDN89xGTYx2Nqa9EYPgNvtfX8aHCTfRZgoi05c/BB7fCoac7Vw/GRJLN0mbCyKqYIunbd+D1KdBluDO/g9UJm1h7dChkdHMfXZ2fLbr8ccgOn0PBWBVVQ2MJIlJ+XgIvT4Q2h8LpzzpTSRoTa40y4Kcv4JuXPRYKND9wX+LwORSMVVE1NJYgIqHgW3jhdEhvC+NnQ3JarCMyxnH2q87Pkt2wfS1s+w62fb/v5w8fxzY+U6tYggi33zbCcydDXCKc9So0aRXriExD0zjTdxVRpaRG0La38/BUUQG3NI9sfKbOsAQRTnt2wPOnOHNKn/smtLDJ2U0M1KSdwNrJjAf7NIRL6R548Uznsv2MGdC2T6wjMib8dv8S6whMFNkVRKh89fRITofOR0U/HmPCxVcVFcD0Y5x2teYHRjcmExOWIELl6x+o2AbjM3WcryqqDZ85050+ORLGv2xXyQ2AVTEZYwKTdQRMehfik+CpY2Hte7GOyESYJQhjTOAye8B5C6B5J5hxOnz5fKwjMhFkCcIYE5z0tnDuW9BpKMy9BBbe6QxIaeodSxDGmOClpDvtEH3OhIX/hNcnO8ONm3rFGqlDFcjNSMbUZ/GJcOLD0LQDfHw37NwMpz4FyU1iHZkJE0sQobJBy4zZN8R4ent480q480BnToqqbKC/OsmqmIwxNZdzLpzxovfkADbQXx1lCcIYEx7dR8c6AhNmEU0QIjJaRL4VkbUiMtXPdqeIiIpIjvs6S0T2iMgK9/FIJOM0xhizv4i1QYhIPPAQMBLIB5aKyDxVXVVluzRgCvBFlV2sU9XsSMVnjImyR4dCvwlw6GlOLyhT60XyCmIAsFZV16tqCTATOMHLdrcCdwJFEYzFGBNrFeVOQ/a93WHOJc6kWnb/RK0WyV5M7YGfPV7nAwM9NxCRfkBHVX1TRK6uUr6TiHwJFAI3qOonVQ8gIucD5wMccMAB4YzdGBMKf92/L/wUNi6H5U/DN6/Aiuchsyf0Owd6j4WHBtpUp7VMzLq5ikgccB8w0cvqTcABqrpdRPoDc0Skl6r+YSQ8VX0MeAwgJyfHvooYE2vVncg79Hcex/wTVr4Cy56Bd6bCgpugvNh7GesBFTORrGLaCHT0eN3BXVYpDTgEWCgiG4BBwDwRyVHVYlXdDqCqy4B1QLcIxmqMiabkNOg/Ec7/0Lmy6HdOrCMyXkQyQSwFuopIJxFJAs4A5lWuVNXfVDVDVbNUNQtYDIxR1VwRaeU2ciMinYGuwPoIxmqMiZU2h8Jx98Q6CuNFxBKEqpYBlwLzgdXAS6qaJyK3iMiYaooPBb4WkRXAbOBCVbWprIxpqGwmu5gQrSe9CHJycjQ3NzfWYRhjQjWtqe91qS3g6GnQ92ybNzvMRGSZquZ4W2fvtDGmdvA10GVqC2jV3Rkxdvoo2PRVdONqwGywPmNM7eCvB5QqfDUTFvwDHhsGh/0V/nQ9pDaLVnQNkl1BGGNqPxHIPhMuzYWc82DpE/BgDqx40W62iyBrgzDG1D3/W+Hclb1xGRwwGArWwB4vDdl2k121rA3CGFO/tMuG896DvzwABau9Jwewm+xqyBKEMaZuiotzbra7dFmsI6m3LEEYY+q2xi39r//8v05VVHlpdOKpR6wXkzGmfpt/nfMzsRG07w8HDIKOg6DjYZDSFO7uaoME+mAJwhhTv125Gn5aDD9/4fz85D7QckCc0WR9tVNY+4UlCGNMPeBvmPH0dnDIyc4DoHgXbMyFn76AnxfD1rzoxlqHWIIwxtR9wVQFJTeBzsOcB/gf4qOBs0ZqY4wxXlmCMMYYXzYuj3UEMWUJwhjTsPkaJFDi4NkTIb/hjtBgbRDGmIbNV/vFrz/DM8fDcyfBWa9AxwHRjasWsCsIY4zxpllHmPgWNM5wksSPn8c6oqizBGGMMb40be8kibQ28PwpsOGzWEcUVVbFZIwx/qS3hYlvwjNjYMapMG4WdBoann3X8ru47QrCGGOqk9YGJr4BzQ6EGafD+oXh2W8tv4vbEoQxxgSiSSZMeB1adIYXxsLa92MdUcRZgjDGmEA1aeUkiZZd4cUz4fsFwZVXhS158OHt8NCgyMQYRtYGYYwxwWjcEibMg+dOhJnj4PTnoPto39urwqavYNVcWD0Ptq8FBA48IloRh8wShDHGBKtRCzhnrtP99cWx3rdJbQ59z3ISw68/gcRDpyEw6GLocTykta7140BZgjDGmFCkNoez58CdB3pfv2cHLH7EGRRw6DXQ/dj9JzfyNQptfLJz5SES7qiDYgnCGGNCldrM//qr1/rfxltX1k/vh/dugs/uhyOvCD22MLBGamOMiZTqEog3R0yBXifDezfD2vfCHlIwLEEYY0xtIgInPOjMdjf7PPhlfcxCsQRhjDG1TVJjOGOG83zmWVDye0zCsARhjDE14Wu4cF/LA9WiE5w6HQpWw9xLnEbrKLNGamOMqYlIjpl00AgYcZPTaN02G468PHLH8sKuIIwxpjY7Ygr0Ognevznqw3tYgjDGmNpMBE54CFodDLMnRbXR2hKEMcbUdjFqtLYEYYwxdUEMGq0tQRhjTF1R2Wid9xp89kDED2cJwhhj6pIoNlpbN1djjKlLKhutV82D50/ef30YpyuN6BWEiIwWkW9FZK2ITPWz3SkioiKS47HsOrfctyJyTCTjNMaYOiWpMWi593VhnK40YlcQIhIPPASMBPKBpSIyT1VXVdkuDZgCfOGxrCdwBtALaAe8JyLdVH29I8YYY8ItklcQA4C1qrpeVUuAmcAJXra7FbgTKPJYdgIwU1WLVfUHYK27P2OMMVESyQTRHvjZ43W+u2wvEekHdFTVN4Mt65Y/X0RyRSS3oKAgPFEbY4wBYtiLSUTigPuAq0Ldh6o+pqo5qprTqlWr8AVnjDEmor2YNgIdPV53cJdVSgMOARaKM61eG2CeiIwJoKwxxjRsvqYrrekosh4imSCWAl1FpBPOyf0MYFzlSlX9DciofC0iC4H/U9VcEdkDvCAi9+E0UncFlkQwVmOMqVsiOYqsK2IJQlXLRORSYD4QD0xX1TwRuQXIVdV5fsrmichLwCqgDLjEejAZY0x0icZgEopIyMnJ0dzc3FiHYYwxdYqILFPVHG/rbKgNY4wxXlmCMMYY45UlCGOMMV7VmzYIESkAfqzBLjKAbQ2obCyPXRfLxvLY9jvXjbKxPHZNyh6oqt5vJFNVezhJMrchla2rcdv7Zb9zbS1bl+P29bAqJmOMMV5ZgjDGGOOVJYh9HmtgZWN57LpYNpbHtt+5bpSN5bFrGrdX9aaR2hhjTHjZFYQxxhivLEEYY4zxqsEniEDnzfZRdrqIbBWRlSEct6OIfCgiq0QkT0SmBFE2RUSWiMhXbtmbQzh+vIh8KSJvBFlug4h8IyIrRCTowa9EpJmIzBaRNSKyWkQOD7Bcd/eYlY9CEbk8iONe4b5XK0XkRRFJCaLsFLdcXiDH9Pa5EJEWIrJARL53fzYPouxp7rErPOdtD7Ds3e57/bWIvCYizYIoe6tbboWIvCsi7YI5tse6q9w55zMCLSsi00Rko8ff+9hgjisil7m/d56I3BXEcWd5HHODiKwIomy2iCyu/N8QEZ+zYPoo30dEPnf/v14XkXQv5byeNwL9fAUtEn1n68oDZ5TZdUBnIAn4CugZRPmhQD9gZQjHbgv0c5+nAd8FemxAgCbu80Sc+bwHBXn8K4EXgDeCLLcByKjBe/4M8Ff3eRLQLMS/22acG3wC2b498AOQ6r5+CZgYYNlDgJVAI5zRj98DDgr2cwHcBUx1n08F7gyi7MFAd2AhkBPkcUcBCe7zO4M8brrH88nAI8Ec213eEWdE5x99fW58HHsazvD/Qf8PAn9y/07J7uvMYGL2WH8vcGMQx30X+LP7/FhgYZBxLwWOcp9PAm71Us7reSPQz1ewj4Z+BRHovNleqerHwC+hHFhVN6nqcvf5TmA1XqZV9VFWVXWX+zLRfQTc20BEOgDHAU8EFXQNiUhTnH+MJwFUtURVfw1hVyOAdaoazJ3zCUCqiCTgnOz/F2C5g4EvVHW3qpYBHwEn+yvg43NxAk5yxP15YqBlVXW1qn5bXaA+yr7rxg2wGGfyrUDLFnq8bIyfz5if/4V/AdeEWLZaPspeBNyhqsXuNl5m1fF/XBER4HTgxSDKKlD5rb8pfj5jPsp3Az52ny8ATvFSztd5I6DPV7AaeoIIaO7rSBORLKAvzpVAoGXi3cvfrcACVQ24LHA/zj9tRRBlKinwrogsE5HzgyzbCSgAnnKrt54QkcYhxHAGPv5xvVHVjcA9wE/AJuA3VX03wOIrgSEi0lJEGuF8M+xYTRlvWqvqJvf5ZqB1CPuoqUnA28EUEJH/JyI/A+OBG4MsewKwUVW/Cqach0vdKq7pQVaZdMP5m30hIh+JyGEhHHsIsEVVg5mV53Lgbvf9uge4Lshj5rHvC+ppVPM5q3LeiMjnq6EniJgTkSbAK8DlVb6x+aWq5aqajfONcICIHBLg8Y4HtqrqslDiBY5U1X7An4FLRGRoEGUTcC6rH1bVvsDvOJfDARORJGAM8HIQZZrj/ON1wpmhsLGInBVIWVVdjVM18y7wDrACqNHkVerUA0S1f7mIXI8z+daMYMqp6vWq2tEtd2kQx2sE/J0gk4qHh4EuQDZOUr83iLIJQAtgEHA18JJ7RRCMMwniS4jrIuAK9/26AvdKOQiTgItFZBlO9VGJrw39nTfC+flq6AkipnNfi0gizh95hqq+Gso+3CqaD4HRARY5AhgjIhtwqtSGi8jzQRxvo/tzK/AaTjVdoPKBfI+rndk4CSMYfwaWq+qWIMocDfygqgWqWgq8CgwOtLCqPqmq/VV1KLADp943WFtEpC2A+9NrtUckiMhE4HhgvHvyCMUMvFR5+NEFJyF/5X7WOgDLRaRNIIVVdYv7JagCeJzgP2evulWxS3CulL02kHvjVkOeDMwK4pgAE3A+W+B8gQkmZlR1jaqOUtX+OMlpnY/4vJ03IvL5augJYu+82e430zMAn1OhhpP7jeZJYLWq3hdk2VaVvVFEJBUYCawJpKyqXqeqHVQ1C+f3/UBVA/o2LSKNRSSt8jlOA2jAPbhUdTPws4h0dxeNwJlWNhihfLP7CRgkIo3c930ETt1tQEQk0/15AM6J44Ugjw/O52qC+3wCMDeEfQRNREbjVCeOUdXdQZbt6vHyBAL8jAGo6jeqmqmqWe5nLR+ncXVzgMdu6/HyJIL4nAFzcBqqEZFuOJ0hghnp9GhgjarmB1EGnDaHo9znw4GgJo32+JzFATcAj3jZxtd5IzKfr3C0dNflB06d8nc42fr6IMu+iHP5W4rzD3BeEGWPxLkM/Bqn2mIFcGyAZXsDX7plV+Kjp0UA+xlGEL2YcHp7feU+8oJ9v9x9ZAO5buxzgOZBlG0MbAeahnDcm3FOcCuB53B7uARY9hOcRPYVMCKUzwXQEngf56TxHtAiiLInuc+LgS3A/CDKrsVpZ6v8jHntieSj7Cvu+/U18DrQPtT/Bfz0fvNx7OeAb9xjzwPaBlE2CXjejX05MDyYmIGngQtD+BsfCSxzPydfAP2DLD8F51z0HXAH7kgXVcp5PW8E+vkK9mFDbRhjjPGqoVcxGWOM8cEShDHGGK8sQRhjjPHKEoQxxhivLEEYY4zxyhKEMT6ISLn8cQTZqe7yheKMAPyViHxWeV+HiCSJyP3ijAz8vYjMdce9qtxfGxGZKSLr3KFK3hKRbiKSJfuPRjpNRP7PfT7IHTZihTgj4E6L4ttgGrCEWAdgTC22R53hTLwZr6q57nhUd+MM//FPnCESuqtquYicC7wqIgPdMq8Bz6jqGeAM74wzZs7P++/+D54BTlfVr0QkHmdkV2MizhKEMTXzMXC5O/bQuUAnVS0HUNWnRGQSzl21CpSq6t67Y9UdxM4ddM2fTJybqnD3Hezd58aExBKEMb6lyh8njLldVauOz/MXnDt+DwJ+0v0HXMwFernP/Q2Q2KXKsdrgjAgKzpDZ34rIQpwBA59R1aJAfwljQmUJwhjf/FUxzRCRPThDSFwG1HQGr3Wex/JsZ1DVW0RkBs7YV+NwxqMaVsPjGVMtSxDGhGa8qu6dclVEfgEOEJE0dSZyqdQfqJzW9dRQD6aq64CHReRxoEBEWqrq9lD3Z0wgrBeTMWGgqr/jNCbf5zYkIyLn4Mxe94H7SPacZElEeovIkOr2LSLHecxn0BVnPopfw/sbGLM/SxDG+JZapZvrHdVsfx1QBHwnIt/jzAp2krpwRmU92u3mmgfcjjP7V3XOxmmDWIEzyun4yoZwYyLJRnM1xhjjlV1BGGOM8coShDHGGK8sQRhjjPHKEoQxxhivLEEYY4zxyhKEMcYYryxBGGOM8er/AxJQFOw0o8vUAAAAAElFTkSuQmCC\\n\",\n      \"text/plain\": [\n       \"<Figure size 432x288 with 1 Axes>\"\n      ]\n     },\n     \"metadata\": {\n      \"needs_background\": \"light\"\n     },\n     \"output_type\": \"display_data\"\n    },\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 208 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"value1 = ft_recall_30\\n\",\n    \"value2 = cs_recall_30\\n\",\n    \"name = \\\"Recall@30\\\"\\n\",\n    \"\\n\",\n    \"length = len(value1)\\n\",\n    \"plt.plot(range(length), value1, \\\"-s\\\", label=\\\"Finetune\\\")\\n\",\n    \"plt.plot(range(1, length), value2, \\\"-s\\\", label=\\\"w/o Pretrain\\\")\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"plt.xticks(range(length))\\n\",\n    \"plt.ylabel(name)\\n\",\n    \"plt.xlabel(\\\"EPOCHS\\\")\\n\",\n    \"plt.legend(loc='center right')\\n\",\n    \"plt.show()\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"united-recall\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  }\n ],\n \"metadata\": {\n  \"kernelspec\": {\n   \"display_name\": \"Python 3\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.7.11\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n"
  },
  {
    "path": "finetune/autoDiag/fine-tune.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"id\": \"wicked-finder\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 205 µs\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"%load_ext autotime\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"id\": \"random-fluid\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 1.51 s\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"import numpy as np\\n\",\n    \"import _pickle as pickle\\n\",\n    \"import pandas as pd\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"id\": \"diverse-vegetation\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"==========LOADING DATA==========\\n\",\n      \"time: 14.1 s\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(\\\"==========LOADING DATA==========\\\")\\n\",\n    \"age_seq = pickle.load(open(\\\"../data/new_age_seq\\\",\\\"rb\\\"))\\n\",\n    \"sex_seq = pickle.load(open(\\\"../data/new_sex_seq\\\",\\\"rb\\\"))\\n\",\n    \"\\n\",\n    \"util_seq = pickle.load(open(\\\"../data/new_util_seq\\\",\\\"rb\\\"))\\n\",\n    \"code_seq = pickle.load(open(\\\"../data/new_code_seq\\\",\\\"rb\\\"))\\n\",\n    \"date_seq = pickle.load(open(\\\"../data/new_date_seq\\\",\\\"rb\\\"))\\n\",\n    \"label_seq = pickle.load(open(\\\"../data/new_label_seq\\\",\\\"rb\\\"))\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"id\": \"judicial-aircraft\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 851 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"k = 10000\\n\",\n    \"age_seq = age_seq[:k]\\n\",\n    \"sex_seq = sex_seq[:k]\\n\",\n    \"\\n\",\n    \"util_seq = util_seq[:k]\\n\",\n    \"code_seq = code_seq[:k]\\n\",\n    \"date_seq = date_seq[:k]\\n\",\n    \"label_seq = label_seq[:k]\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"id\": \"disabled-favor\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 3.5 s\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"import numpy as np\\n\",\n    \"import tensorflow as tf\\n\",\n    \"from tensorflow.keras import layers\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"id\": \"hollow-environment\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 10.5 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"class DataGenerator(tf.keras.utils.Sequence):\\n\",\n    \"    def __init__(self, seqs, vocab_sizes, list_IDs, max_visit, max_code, batch_size=100, shuffle=True):\\n\",\n    \"        self.seqs = seqs\\n\",\n    \"        self.code_vocab = vocab_sizes[0]\\n\",\n    \"        self.cat_vocab = vocab_sizes[1]\\n\",\n    \"        self.list_IDs = list_IDs\\n\",\n    \"        self.max_visit = max_visit\\n\",\n    \"        self.max_code = max_code\\n\",\n    \"        self.batch_size = batch_size\\n\",\n    \"        self.shuffle = shuffle\\n\",\n    \"        self.on_epoch_end()\\n\",\n    \"\\n\",\n    \"    def __len__(self):\\n\",\n    \"        'Denotes the number of batches per epoch'\\n\",\n    \"        return int(np.ceil(len(self.list_IDs) / self.batch_size))\\n\",\n    \"\\n\",\n    \"    def __getitem__(self, index):\\n\",\n    \"        'Generate one batch of data'\\n\",\n    \"        indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]\\n\",\n    \"        list_IDs_temp = [self.list_IDs[k] for k in indexes]\\n\",\n    \"        X, y = self.__data_generation(list_IDs_temp)\\n\",\n    \"        return X, y\\n\",\n    \"\\n\",\n    \"    def on_epoch_end(self):\\n\",\n    \"        'Updates indexes after each epoch' \\n\",\n    \"        self.indexes = np.arange(len(self.list_IDs))\\n\",\n    \"        if self.shuffle == True:\\n\",\n    \"            np.random.shuffle(self.indexes)\\n\",\n    \"\\n\",\n    \"    def __data_generation(self, list_IDs_temp):\\n\",\n    \"        'Generates data containing batch_size samples' \\n\",\n    \"        demo_feature, code_feature, util_feature, date_feature, cat_feature = self.seqs\\n\",\n    \"        batch_demo, batch_code, batch_util, batch_date, batch_cat = [], [], [], [], [] \\n\",\n    \"        for i, ID in enumerate(list_IDs_temp):\\n\",\n    \"            batch_demo.append(demo_feature[ID])\\n\",\n    \"            batch_code.append(code_feature[ID])\\n\",\n    \"            batch_util.append(util_feature[ID])\\n\",\n    \"            batch_date.append(date_feature[ID])\\n\",\n    \"            batch_cat.append(cat_feature[ID])\\n\",\n    \"        \\n\",\n    \"        batch_demo_feature = np.array(batch_demo)\\n\",\n    \"        batch_code_feature = self.code_padding(batch_code)\\n\",\n    \"        batch_util_feature = self.date_padding(batch_util)\\n\",\n    \"        batch_date_feature = self.date_padding(batch_date)\\n\",\n    \"        \\n\",\n    \"        batch_code_label = self.code_labelling(batch_cat)  # predict cat instead\\n\",\n    \"        \\n\",\n    \"        dic = (\\n\",\n    \"            {\\n\",\n    \"                'demo_feature': batch_demo_feature,\\n\",\n    \"                'code_feature': batch_code_feature,\\n\",\n    \"                'util_feature': batch_util_feature,\\n\",\n    \"                'date_feature': batch_date_feature,\\n\",\n    \"            },\\n\",\n    \"            {\\n\",\n    \"                'code_label': batch_code_label,\\n\",\n    \"            })\\n\",\n    \"        return dic\\n\",\n    \"    \\n\",\n    \"    def date_padding(self, seq):\\n\",\n    \"        seq = [x[:-1] for x in seq]\\n\",\n    \"        \\n\",\n    \"        pad_seq = np.zeros((len(seq), self.max_visit))\\n\",\n    \"        for i, p in enumerate(seq):\\n\",\n    \"            pad_seq[i][:len(p)] = p[:self.max_visit]\\n\",\n    \"        return pad_seq\\n\",\n    \"    \\n\",\n    \"    def code_padding(self, seq):\\n\",\n    \"        seq = [x[:-1] for x in seq]\\n\",\n    \"        \\n\",\n    \"        X = np.zeros((len(seq), self.max_visit, self.max_code))\\n\",\n    \"        for i, p in enumerate(seq):\\n\",\n    \"            if len(p) > self.max_visit: \\n\",\n    \"                p = p[:self.max_visit]\\n\",\n    \"            for j, claim in enumerate(p):\\n\",\n    \"                claim = claim[:self.max_code]\\n\",\n    \"                X[i][j][:len(claim)] = claim\\n\",\n    \"        return X\\n\",\n    \"    \\n\",\n    \"    def code_labelling(self, seq):\\n\",\n    \"        seq = [x[-1] for x in seq]\\n\",\n    \"        \\n\",\n    \"        X = np.zeros((len(seq), self.cat_vocab))\\n\",\n    \"        for i, claim in enumerate(seq):\\n\",\n    \"            for c in claim:\\n\",\n    \"                X[i][c] = 1\\n\",\n    \"        return X\\n\",\n    \"\\n\",\n    \"    def cat_labelling(self, seq):\\n\",\n    \"        seq = [x[:-1] for x in seq]\\n\",\n    \"        \\n\",\n    \"        X = np.zeros((len(seq), self.max_visit, self.cat_vocab))\\n\",\n    \"        for i, p in enumerate(seq):\\n\",\n    \"            if len(p) > self.max_visit: \\n\",\n    \"                p = p[:self.max_visit]\\n\",\n    \"            for j, claim in enumerate(p):\\n\",\n    \"                for c in claim:\\n\",\n    \"                    X[i][j][c] = 1\\n\",\n    \"        return X\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 61,\n   \"id\": \"quick-webster\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 15.8 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"def create_code_mask(code_seq):\\n\",\n    \"    code_mask = tf.cast(tf.math.not_equal(code_seq, 0), tf.float32)\\n\",\n    \"    return code_mask[:,:,:,tf.newaxis]\\n\",\n    \"\\n\",\n    \"def create_visit_mask(seq):\\n\",\n    \"    visit_mask = tf.cast(tf.math.not_equal(seq, 0), tf.float32)\\n\",\n    \"    return visit_mask[:,:]\\n\",\n    \"\\n\",\n    \"def scaled_dot_product_attention(Q, K, V, Q_masks, K_masks):\\n\",\n    \"    d_k = K.get_shape().as_list()[-1] # d_model/h\\n\",\n    \"\\n\",\n    \"    outputs = tf.matmul(Q, tf.transpose(K, [0, 2, 1]))  # (h*N, T_q, T_k)\\n\",\n    \"    outputs /= d_k ** 0.5\\n\",\n    \"\\n\",\n    \"    padding_num = -1e+7\\n\",\n    \"    K_masks = tf.expand_dims(K_masks, 1) # (h*N, 1, T_k)\\n\",\n    \"    K_masks = tf.tile(K_masks, [1, tf.shape(Q)[1], 1]) # (h*N, T_q, T_k)\\n\",\n    \"    paddings = tf.ones_like(outputs) * padding_num\\n\",\n    \"    outputs = tf.where(tf.equal(K_masks, 0), paddings, outputs)  # (h*N, T_q, T_k)\\n\",\n    \"\\n\",\n    \"    outputs = tf.nn.softmax(outputs)\\n\",\n    \"    Q_masks = tf.expand_dims(Q_masks, -1) # (h*N, T_q, 1)\\n\",\n    \"    Q_masks = tf.tile(Q_masks, [1, 1, tf.shape(K)[1]]) # (h*N, T_q, T_k)\\n\",\n    \"    outputs = outputs * tf.cast(Q_masks, dtype=tf.float32)\\n\",\n    \"\\n\",\n    \"    return tf.matmul(outputs, V) # [h*N, T_q, d_model/h]\\n\",\n    \"\\n\",\n    \"class multihead_attention(tf.keras.layers.Layer):\\n\",\n    \"    def __init__(self, d_model, num_heads, name=\\\"multihead_attention\\\"):\\n\",\n    \"        super(multihead_attention, self).__init__(name=name)\\n\",\n    \"        self.num_heads = num_heads\\n\",\n    \"        self.d_model = d_model\\n\",\n    \"\\n\",\n    \"        assert d_model % self.num_heads == 0\\n\",\n    \"\\n\",\n    \"        self.query_dense = layers.Dense(units=d_model, use_bias=False)\\n\",\n    \"        self.key_dense = layers.Dense(units=d_model, use_bias=False)\\n\",\n    \"        self.value_dense = layers.Dense(units=d_model, use_bias=False)\\n\",\n    \"        self.add =layers.Add()\\n\",\n    \"        self.norm = layers.LayerNormalization()\\n\",\n    \"    \\n\",\n    \"    def call(self, queries, keys, values, query_masks, key_masks):\\n\",\n    \"        Q = self.query_dense(queries)\\n\",\n    \"        K = self.key_dense(keys)\\n\",\n    \"        V = self.value_dense(values)\\n\",\n    \"\\n\",\n    \"        # Split and concat\\n\",\n    \"        Q_ = tf.concat(tf.split(Q, self.num_heads, axis=2), axis=0) # (h*N, T_q, d_model/h)\\n\",\n    \"        K_ = tf.concat(tf.split(K, self.num_heads, axis=2), axis=0) # (h*N, T_k, d_model/h)\\n\",\n    \"        V_ = tf.concat(tf.split(V, self.num_heads, axis=2), axis=0) # (h*N, T_v, d_model/h)\\n\",\n    \"        query_masks = tf.tile(query_masks, [self.num_heads, 1]) # (h*N, T_q)\\n\",\n    \"        key_masks = tf.tile(key_masks, [self.num_heads, 1]) # (h*N, T_k)\\n\",\n    \"\\n\",\n    \"        # Attention\\n\",\n    \"        outputs = scaled_dot_product_attention(Q_, K_, V_, query_masks, key_masks) # (h*N, T_q, d_model/h)\\n\",\n    \"\\n\",\n    \"        # Restore shape\\n\",\n    \"        outputs = tf.concat(tf.split(outputs, self.num_heads, axis=0), axis=2) # (N, T_q, d_model)\\n\",\n    \"\\n\",\n    \"        # Residual connection\\n\",\n    \"        outputs = self.add([queries, outputs])\\n\",\n    \"        outputs = self.norm(outputs)\\n\",\n    \"        \\n\",\n    \"        return outputs\\n\",\n    \"\\n\",\n    \"class ffn(tf.keras.layers.Layer):\\n\",\n    \"    def __init__(self, d_model, ffn_dim, name=\\\"ffn\\\"):\\n\",\n    \"        super(ffn, self).__init__(name=name)\\n\",\n    \"        self.ffn_dim = ffn_dim\\n\",\n    \"        self.dense1 = layers.Dense(units=ffn_dim, activation=tf.nn.relu, use_bias=False)\\n\",\n    \"        self.dense2 = layers.Dense(units=d_model, use_bias=False)\\n\",\n    \"        self.add =layers.Add()\\n\",\n    \"        self.norm = layers.LayerNormalization()\\n\",\n    \"    \\n\",\n    \"    def call(self, inputs):\\n\",\n    \"        outputs = self.dense1(inputs)\\n\",\n    \"        outputs = self.dense2(outputs)\\n\",\n    \"        outputs = self.add([inputs, outputs])\\n\",\n    \"        outputs = self.norm(outputs)\\n\",\n    \"        return outputs\\n\",\n    \"\\n\",\n    \"def cat_recall(y_true, y_pred):\\n\",\n    \"    mask_value = tf.cast(tf.not_equal(tf.reduce_sum(y_true,axis=-1), 0), tf.float32)\\n\",\n    \"    true_positives = tf.cast(tf.reduce_sum(tf.multiply(tf.round(y_pred), y_true), axis=-1), tf.float32)\\n\",\n    \"    possible_positives = tf.cast(tf.reduce_sum(y_true, axis=-1), tf.float32)\\n\",\n    \"    values = true_positives / (possible_positives + 1e-7)\\n\",\n    \"    return tf.reduce_sum(values)/tf.reduce_sum(mask_value)\\n\",\n    \"\\n\",\n    \"def cat_loss_fun(y_true, y_pred):\\n\",\n    \"    loss = tf.cast(tf.keras.losses.BinaryCrossentropy(reduction='none')(y_true, y_pred), tf.float32)\\n\",\n    \"    mask = tf.cast(tf.not_equal(tf.reduce_sum(y_true,axis=-1), 0), tf.float32)\\n\",\n    \"    loss = tf.multiply(loss, mask)\\n\",\n    \"    # return tf.reduce_sum(loss)/tf.reduce_sum(mask)\\n\",\n    \"    return loss\\n\",\n    \"\\n\",\n    \"def model(\\n\",\n    \"    max_visit,\\n\",\n    \"    max_code,\\n\",\n    \"    max_demo,\\n\",\n    \"    \\n\",\n    \"    demo_vocab,\\n\",\n    \"    code_vocab,\\n\",\n    \"    date_vocab,\\n\",\n    \"    util_vocab,\\n\",\n    \"    cat_vocab,\\n\",\n    \"\\n\",\n    \"    patient_dim,\\n\",\n    \"    vocab_dim=100,\\n\",\n    \"    model_dim=100,\\n\",\n    \"    ffn_dim=100,\\n\",\n    \"    num_heads=2,\\n\",\n    \"    num_translayer=1,\\n\",\n    \"    \\n\",\n    \"    model_name=\\\"TransF\\\"):\\n\",\n    \"    \\n\",\n    \"    demo = layers.Input(shape=(max_demo, ), name=\\\"demo_feature\\\")  # max_demo = 2, age&sex\\n\",\n    \"    code_seq = layers.Input(shape=(max_visit, max_code), name=\\\"code_feature\\\") \\n\",\n    \"    util_seq = layers.Input(shape=(max_visit), name=\\\"util_feature\\\")\\n\",\n    \"    date_seq = layers.Input(shape=(max_visit), name=\\\"date_feature\\\")\\n\",\n    \"\\n\",\n    \"    inputs = [demo, code_seq, util_seq, date_seq]\\n\",\n    \"    \\n\",\n    \"    # demo embedding\\n\",\n    \"    demo_emb = layers.Embedding(input_dim=demo_vocab, output_dim=vocab_dim, mask_zero=True, name='demo_embedding')(demo)\\n\",\n    \"    demo_emb = layers.Lambda(lambda x: tf.keras.backend.sum(x, axis=1))(demo_emb)     \\n\",\n    \"\\n\",\n    \"    # code sequence\\n\",\n    \"    code_mask = layers.Lambda(create_code_mask)(code_seq)\\n\",\n    \"    code_emb = layers.Embedding(input_dim=code_vocab, \\n\",\n    \"                                output_dim=vocab_dim, \\n\",\n    \"                                name='code_embed')(code_seq)\\n\",\n    \"    code_emb = layers.Multiply()([code_emb, code_mask])\\n\",\n    \"    code_emb = tf.reduce_sum(code_emb, axis=2)     \\n\",\n    \"\\n\",\n    \"    \\n\",\n    \"    # visit mask\\n\",\n    \"    visit_mask = layers.Lambda(create_visit_mask)(date_seq)\\n\",\n    \"    \\n\",\n    \"    # util sequence   \\n\",\n    \"    util_emb = layers.Embedding(input_dim=util_vocab, output_dim=vocab_dim, mask_zero=True, name='util_embedding')(util_seq)\\n\",\n    \"    util_emb = layers.Multiply()([util_emb, visit_mask[:,:,tf.newaxis]])\\n\",\n    \"    \\n\",\n    \"    # date sequence    \\n\",\n    \"    date_emb = layers.Embedding(input_dim=date_vocab, output_dim=vocab_dim, mask_zero=True, name='date_embedding')(date_seq)\\n\",\n    \"    date_emb = layers.Multiply()([date_emb, visit_mask[:,:,tf.newaxis]])\\n\",\n    \"\\n\",\n    \"    # visit sequence\\n\",\n    \"    visit_emb = layers.Add()([code_emb, date_emb, util_emb]) \\n\",\n    \"\\n\",\n    \"    demo_emb = tf.expand_dims(demo_emb, 1) # (N, 1, emb_size)\\n\",\n    \"    demo_mask = tf.ones_like(tf.reduce_sum(demo_emb, axis=2), tf.float32) # (N, 1)\\n\",\n    \"    \\n\",\n    \"    for trans_layer in range(num_translayer):\\n\",\n    \"        multihead = multihead_attention(model_dim, num_heads, name=\\\"multihead_attention-\\\"+str(trans_layer))(\\n\",\n    \"            queries=tf.concat([demo_emb, visit_emb], 1),                         \\n\",\n    \"            keys=tf.concat([demo_emb, visit_emb], 1),\\n\",\n    \"            values=tf.concat([demo_emb, visit_emb], 1),\\n\",\n    \"            query_masks=tf.concat([demo_mask, visit_mask], 1),\\n\",\n    \"            key_masks=tf.concat([demo_mask, visit_mask], 1)\\n\",\n    \"        )\\n\",\n    \"        \\n\",\n    \"        visit_emb = ffn(model_dim, ffn_dim, name=\\\"ffn-\\\"+str(trans_layer))(multihead) # (N, max_visit, emb_size)\\n\",\n    \"            \\n\",\n    \"        demo_emb = visit_emb[:, :1, :] # (N, 1, emb_size)\\n\",\n    \"        visit_emb = visit_emb[:, 1:max_visit+1, :] # (N, max_visit, emb_size)\\n\",\n    \"        \\n\",\n    \"    patient_embedding = layers.Dense(patient_dim, activation=None, name=\\\"patient_embedding\\\")(tf.squeeze(demo_emb, [1]))\\n\",\n    \"    \\n\",\n    \"    # code_label = layers.Dense(units=model_dim, activation=tf.nn.sigmoid)(patient_embedding)\\n\",\n    \"    code_label = layers.Dense(units=cat_vocab, activation=tf.nn.sigmoid, name='code_label')(patient_embedding)\\n\",\n    \"    \\n\",\n    \"    # cat_label = layers.Dense(units=model_dim, activation=tf.nn.sigmoid)(visit_emb)\\n\",\n    \"    # cat_label = layers.Dense(units=cat_vocab, activation=tf.nn.sigmoid, name='cat_label')(visit_emb)\\n\",\n    \"    \\n\",\n    \"    outputs = [code_label]\\n\",\n    \"    return tf.keras.Model(inputs=inputs, outputs=outputs, name=model_name)\\n\",\n    \"\\n\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 8,\n   \"id\": \"dependent-movie\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 6.54 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"def process_code(seq, vocab2int):\\n\",\n    \"    unseen = []\\n\",\n    \"    new_seq = []\\n\",\n    \"    for p in seq:\\n\",\n    \"        new_p = []\\n\",\n    \"        for v in p:\\n\",\n    \"            new_v = []\\n\",\n    \"            for c in v:\\n\",\n    \"                if c not in vocab2int: \\n\",\n    \"                    unseen.append(c)\\n\",\n    \"                    continue\\n\",\n    \"                    # vocab2int[c] = len(vocab2int)\\n\",\n    \"                new_v.append(vocab2int[c])\\n\",\n    \"            new_p.append(new_v)\\n\",\n    \"        new_seq.append(new_p)\\n\",\n    \"        \\n\",\n    \"    print(\\\"UNSEEN VOCAB:\\\",len(set(unseen)), len(unseen))\\n\",\n    \"    return new_seq\\n\",\n    \"\\n\",\n    \"def process_util(seq, util2int):\\n\",\n    \"    new_seq = []\\n\",\n    \"    vocab2int = {\\\"PAD\\\":0,\\\"IP\\\":1,\\\"RX\\\":2,\\\"OP\\\":3}\\n\",\n    \"    for p in seq:\\n\",\n    \"        new_p = []\\n\",\n    \"        for v in p:\\n\",\n    \"            if \\\"IP\\\" in v:\\n\",\n    \"                new_v=1\\n\",\n    \"            elif \\\"RX\\\" in v:\\n\",\n    \"                new_v=2\\n\",\n    \"            else:\\n\",\n    \"                new_v=3\\n\",\n    \"            new_p.append(new_v)\\n\",\n    \"        new_seq.append(new_p)\\n\",\n    \"    return new_seq\\n\",\n    \"    \\n\",\n    \"def process_demo(age_seq, sex_seq, vocab2int):\\n\",\n    \"    new_seq = []\\n\",\n    \"    for age, sex in zip(age_seq, sex_seq):\\n\",\n    \"        p = []\\n\",\n    \"        assert age in vocab2int\\n\",\n    \"        assert sex in vocab2int\\n\",\n    \"        \\n\",\n    \"        p.append(vocab2int[age])\\n\",\n    \"        p.append(vocab2int[sex])\\n\",\n    \"        new_seq.append(p)\\n\",\n    \"    return np.array(new_seq)\\n\",\n    \"\\n\",\n    \"def get_cat(seq,code2cat):\\n\",\n    \"    new_seq = []\\n\",\n    \"    for p in seq:\\n\",\n    \"        new_p = []\\n\",\n    \"        for v in p:\\n\",\n    \"            new_v = []\\n\",\n    \"            for c in v:\\n\",\n    \"                new_c = code2cat[c]\\n\",\n    \"                new_v.append(new_c)\\n\",\n    \"            new_p.append(new_v)\\n\",\n    \"        new_seq.append(new_p)\\n\",\n    \"    return new_seq\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 9,\n   \"id\": \"ongoing-manufacturer\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"------LOADING DIC------\\n\",\n      \"time: 83.4 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(\\\"------LOADING DIC------\\\")\\n\",\n    \"path = \\\"/Users/xxz005/Desktop/RAW_DATA/code2cat/\\\"\\n\",\n    \"\\n\",\n    \"diag2cat = pickle.load(open(path+\\\"diag2cat\\\",\\\"rb\\\"))\\n\",\n    \"proc2cat = pickle.load(open(path+\\\"proc2cat\\\",\\\"rb\\\"))\\n\",\n    \"drug2cat = pickle.load(open(path+\\\"drug2cat\\\",\\\"rb\\\"))\\n\",\n    \"\\n\",\n    \"code2cat = {**diag2cat, **proc2cat, **drug2cat}\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 10,\n   \"id\": \"wrong-warrior\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 34 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"code2int, util2int, demo2int, cat2int  = pickle.load(open(\\\"../../pretraining/model/saveModel/vocabs\\\",\\\"rb\\\"))\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 11,\n   \"id\": \"invalid-silicon\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"UNSEEN VOCAB: 0 0\\n\",\n      \"UNSEEN VOCAB: 0 0\\n\",\n      \"time: 667 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"code_feature = process_code(code_seq, code2int)\\n\",\n    \"util_feature = process_util(util_seq, util2int)\\n\",\n    \"demo_feature = process_demo(age_seq, sex_seq, demo2int)\\n\",\n    \"\\n\",\n    \"date_feature = date_seq\\n\",\n    \"\\n\",\n    \"cat_seq = get_cat(code_seq, code2cat)\\n\",\n    \"cat_feature = process_code(cat_seq, cat2int)  \"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 35,\n   \"id\": \"adverse-arkansas\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 1.27 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"MAX_VISIT=30\\n\",\n    \"MAX_CODE=10\\n\",\n    \"MAX_DEMO=2\\n\",\n    \"PATIENT_DIM=100\\n\",\n    \"\\n\",\n    \"BATCH_SIZE = 500\\n\",\n    \"TRAIN_RATIO = 0.7\\n\",\n    \"DATA_SIZE = len(age_seq)\\n\",\n    \"EPOCHS = 20\\n\",\n    \"\\n\",\n    \"params = {\\n\",\n    \"    'seqs':[demo_feature, code_feature, util_feature, date_feature, cat_feature],\\n\",\n    \"    'vocab_sizes': [len(code2int), len(cat2int)],\\n\",\n    \"    'batch_size':100,\\n\",\n    \"    'max_visit':MAX_VISIT, \\n\",\n    \"    'max_code':MAX_CODE,\\n\",\n    \"}\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 36,\n   \"id\": \"sorted-wholesale\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 7.39 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"from sklearn.model_selection import train_test_split\\n\",\n    \"train_IDs, valid_IDs = train_test_split(range(DATA_SIZE), train_size=TRAIN_RATIO, random_state=42)\\n\",\n    \"train_generator = DataGenerator(list_IDs=train_IDs, shuffle=True, **params)\\n\",\n    \"valid_generator = DataGenerator(list_IDs=valid_IDs, shuffle=False, **params)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"separated-layout\",\n   \"metadata\": {},\n   \"source\": [\n    \"# Pretrain\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 56,\n   \"id\": \"suspended-settlement\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 2.58 s\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"model_path = \\\"../../pretraining/model/saveModel\\\"\\n\",\n    \"\\n\",\n    \"model = tf.keras.models.load_model(model_path)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 57,\n   \"id\": \"veterinary-chinese\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Model: \\\"TransF\\\"\\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"Layer (type)                    Output Shape         Param #     Connected to                     \\n\",\n      \"==================================================================================================\\n\",\n      \"demo_feature (InputLayer)       [(None, 2)]          0                                            \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"demo_embedding (Embedding)      (None, 2, 100)       2400        demo_feature[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"lambda (Lambda)                 (None, 100)          0           demo_embedding[0][0]             \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"code_feature (InputLayer)       [(None, 30, 10)]     0                                            \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"date_feature (InputLayer)       [(None, 30)]         0                                            \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_ExpandDims (TensorF (None, 1, 100)       0           lambda[0][0]                     \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"code_embed (Embedding)          (None, 30, 10, 100)  4983600     code_feature[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"lambda_1 (Lambda)               (None, 30, 10, 1)    0           code_feature[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"lambda_2 (Lambda)               (None, 30)           0           date_feature[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"util_feature (InputLayer)       [(None, 30)]         0                                            \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"multiply (Multiply)             (None, 30, 10, 100)  0           code_embed[0][0]                 \\n\",\n      \"                                                                 lambda_1[0][0]                   \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"date_embedding (Embedding)      (None, 30, 100)      36600       date_feature[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_strided_slice_1 (Te (None, 30, 1)        0           lambda_2[0][0]                   \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"util_embedding (Embedding)      (None, 30, 100)      400         util_feature[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_strided_slice (Tens (None, 30, 1)        0           lambda_2[0][0]                   \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_Sum_1 (TensorFlowOp (None, 1)            0           tf_op_layer_ExpandDims[0][0]     \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_Sum (TensorFlowOpLa (None, 30, 100)      0           multiply[0][0]                   \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"multiply_2 (Multiply)           (None, 30, 100)      0           date_embedding[0][0]             \\n\",\n      \"                                                                 tf_op_layer_strided_slice_1[0][0]\\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"multiply_1 (Multiply)           (None, 30, 100)      0           util_embedding[0][0]             \\n\",\n      \"                                                                 tf_op_layer_strided_slice[0][0]  \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_Shape (TensorFlowOp (2,)                 0           tf_op_layer_Sum_1[0][0]          \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"add (Add)                       (None, 30, 100)      0           tf_op_layer_Sum[0][0]            \\n\",\n      \"                                                                 multiply_2[0][0]                 \\n\",\n      \"                                                                 multiply_1[0][0]                 \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_Fill (TensorFlowOpL (None, 1)            0           tf_op_layer_Shape[0][0]          \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_concat (TensorFlowO (None, 31, 100)      0           tf_op_layer_ExpandDims[0][0]     \\n\",\n      \"                                                                 add[0][0]                        \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_concat_4 (TensorFlo (None, 31)           0           tf_op_layer_Fill[0][0]           \\n\",\n      \"                                                                 lambda_2[0][0]                   \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_concat_1 (TensorFlo (None, 31, 100)      0           tf_op_layer_ExpandDims[0][0]     \\n\",\n      \"                                                                 add[0][0]                        \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_concat_3 (TensorFlo (None, 31)           0           tf_op_layer_Fill[0][0]           \\n\",\n      \"                                                                 lambda_2[0][0]                   \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_concat_2 (TensorFlo (None, 31, 100)      0           tf_op_layer_ExpandDims[0][0]     \\n\",\n      \"                                                                 add[0][0]                        \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"multihead_attention-0 (multihea (None, 31, 100)      30200       tf_op_layer_concat[0][0]         \\n\",\n      \"                                                                 tf_op_layer_concat_4[0][0]       \\n\",\n      \"                                                                 tf_op_layer_concat_1[0][0]       \\n\",\n      \"                                                                 tf_op_layer_concat_3[0][0]       \\n\",\n      \"                                                                 tf_op_layer_concat_2[0][0]       \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"ffn-0 (ffn)                     (None, 31, 100)      20200       multihead_attention-0[0][0]      \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_strided_slice_2 (Te (None, 1, 100)       0           ffn-0[0][0]                      \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_Squeeze (TensorFlow (None, 100)          0           tf_op_layer_strided_slice_2[0][0]\\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"patient_embedding (Dense)       (None, 100)          10100       tf_op_layer_Squeeze[0][0]        \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_strided_slice_3 (Te (None, 30, 100)      0           ffn-0[0][0]                      \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"code_label (Dense)              (None, 3357)         339057      patient_embedding[0][0]          \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"cat_label (Dense)               (None, 30, 3357)     339057      tf_op_layer_strided_slice_3[0][0]\\n\",\n      \"==================================================================================================\\n\",\n      \"Total params: 5,761,614\\n\",\n      \"Trainable params: 5,761,614\\n\",\n      \"Non-trainable params: 0\\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"None\\n\",\n      \"time: 41.9 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"opt = tf.keras.optimizers.Adam(learning_rate=0.0001)\\n\",\n    \"model.compile(optimizer=opt, loss=tf.keras.losses.BinaryCrossentropy(), metrics=[tf.keras.metrics.Recall(top_k=5), \\n\",\n    \"                                                                                       tf.keras.metrics.Recall(top_k=10), \\n\",\n    \"                                                                                       tf.keras.metrics.Recall(top_k=30)])\\n\",\n    \"print(model.summary())\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 58,\n   \"id\": \"vanilla-karen\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"30/30 [==============================] - 3s 61ms/step - loss: 0.0046 - code_label_loss: 0.0046 - cat_label_loss: 0.0000e+00 - code_label_recall_19: 0.2107 - code_label_recall_20: 0.3366 - code_label_recall_21: 0.5424 - cat_label_recall_19: 0.0000e+00 - cat_label_recall_20: 0.0000e+00 - cat_label_recall_21: 0.0000e+00\\n\"\n     ]\n    },\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"[0.004690711852163076,\\n\",\n       \" 0.004690711852163076,\\n\",\n       \" 0.0,\\n\",\n       \" 0.2135988175868988,\\n\",\n       \" 0.3411019444465637,\\n\",\n       \" 0.5410114526748657,\\n\",\n       \" 0.0,\\n\",\n       \" 0.0,\\n\",\n       \" 0.0]\"\n      ]\n     },\n     \"execution_count\": 58,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    },\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 2.81 s\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"model.evaluate(valid_generator)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 59,\n   \"id\": \"verbal-kingdom\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Epoch 1/20\\n\",\n      \"WARNING:tensorflow:Gradients do not exist for variables ['cat_label/kernel:0', 'cat_label/bias:0'] when minimizing the loss.\\n\",\n      \"WARNING:tensorflow:Gradients do not exist for variables ['cat_label/kernel:0', 'cat_label/bias:0'] when minimizing the loss.\\n\",\n      \"70/70 - 14s - loss: 0.0044 - code_label_loss: 0.0044 - cat_label_loss: 0.0000e+00 - code_label_recall_19: 0.2476 - code_label_recall_20: 0.3654 - code_label_recall_21: 0.5521 - cat_label_recall_19: 0.0000e+00 - cat_label_recall_20: 0.0000e+00 - cat_label_recall_21: 0.0000e+00 - val_loss: 0.0043 - val_code_label_loss: 0.0043 - val_cat_label_loss: 0.0000e+00 - val_code_label_recall_19: 0.2744 - val_code_label_recall_20: 0.3807 - val_code_label_recall_21: 0.5642 - val_cat_label_recall_19: 0.0000e+00 - val_cat_label_recall_20: 0.0000e+00 - val_cat_label_recall_21: 0.0000e+00\\n\",\n      \"Epoch 2/20\\n\",\n      \"70/70 - 10s - loss: 0.0042 - code_label_loss: 0.0042 - cat_label_loss: 0.0000e+00 - code_label_recall_19: 0.2860 - code_label_recall_20: 0.3921 - code_label_recall_21: 0.5814 - cat_label_recall_19: 0.0000e+00 - cat_label_recall_20: 0.0000e+00 - cat_label_recall_21: 0.0000e+00 - val_loss: 0.0042 - val_code_label_loss: 0.0042 - val_cat_label_loss: 0.0000e+00 - val_code_label_recall_19: 0.2876 - val_code_label_recall_20: 0.3942 - val_code_label_recall_21: 0.5826 - val_cat_label_recall_19: 0.0000e+00 - val_cat_label_recall_20: 0.0000e+00 - val_cat_label_recall_21: 0.0000e+00\\n\",\n      \"Epoch 3/20\\n\",\n      \"70/70 - 10s - loss: 0.0041 - code_label_loss: 0.0041 - cat_label_loss: 0.0000e+00 - code_label_recall_19: 0.2990 - code_label_recall_20: 0.4049 - code_label_recall_21: 0.6000 - cat_label_recall_19: 0.0000e+00 - cat_label_recall_20: 0.0000e+00 - cat_label_recall_21: 0.0000e+00 - val_loss: 0.0041 - val_code_label_loss: 0.0041 - val_cat_label_loss: 0.0000e+00 - val_code_label_recall_19: 0.2922 - val_code_label_recall_20: 0.4011 - val_code_label_recall_21: 0.5971 - val_cat_label_recall_19: 0.0000e+00 - val_cat_label_recall_20: 0.0000e+00 - val_cat_label_recall_21: 0.0000e+00\\n\",\n      \"Epoch 4/20\\n\",\n      \"70/70 - 10s - loss: 0.0040 - code_label_loss: 0.0040 - cat_label_loss: 0.0000e+00 - code_label_recall_19: 0.3072 - code_label_recall_20: 0.4153 - code_label_recall_21: 0.6133 - cat_label_recall_19: 0.0000e+00 - cat_label_recall_20: 0.0000e+00 - cat_label_recall_21: 0.0000e+00 - val_loss: 0.0041 - val_code_label_loss: 0.0041 - val_cat_label_loss: 0.0000e+00 - val_code_label_recall_19: 0.2964 - val_code_label_recall_20: 0.4052 - val_code_label_recall_21: 0.6013 - val_cat_label_recall_19: 0.0000e+00 - val_cat_label_recall_20: 0.0000e+00 - val_cat_label_recall_21: 0.0000e+00\\n\",\n      \"Epoch 5/20\\n\",\n      \"70/70 - 10s - loss: 0.0039 - code_label_loss: 0.0039 - cat_label_loss: 0.0000e+00 - code_label_recall_19: 0.3150 - code_label_recall_20: 0.4223 - code_label_recall_21: 0.6242 - cat_label_recall_19: 0.0000e+00 - cat_label_recall_20: 0.0000e+00 - cat_label_recall_21: 0.0000e+00 - val_loss: 0.0040 - val_code_label_loss: 0.0040 - val_cat_label_loss: 0.0000e+00 - val_code_label_recall_19: 0.3007 - val_code_label_recall_20: 0.4090 - val_code_label_recall_21: 0.6083 - val_cat_label_recall_19: 0.0000e+00 - val_cat_label_recall_20: 0.0000e+00 - val_cat_label_recall_21: 0.0000e+00\\n\",\n      \"Epoch 6/20\\n\",\n      \"70/70 - 10s - loss: 0.0039 - code_label_loss: 0.0039 - cat_label_loss: 0.0000e+00 - code_label_recall_19: 0.3216 - code_label_recall_20: 0.4308 - code_label_recall_21: 0.6324 - cat_label_recall_19: 0.0000e+00 - cat_label_recall_20: 0.0000e+00 - cat_label_recall_21: 0.0000e+00 - val_loss: 0.0040 - val_code_label_loss: 0.0040 - val_cat_label_loss: 0.0000e+00 - val_code_label_recall_19: 0.3042 - val_code_label_recall_20: 0.4122 - val_code_label_recall_21: 0.6110 - val_cat_label_recall_19: 0.0000e+00 - val_cat_label_recall_20: 0.0000e+00 - val_cat_label_recall_21: 0.0000e+00\\n\",\n      \"Epoch 7/20\\n\",\n      \"70/70 - 10s - loss: 0.0038 - code_label_loss: 0.0038 - cat_label_loss: 0.0000e+00 - code_label_recall_19: 0.3261 - code_label_recall_20: 0.4373 - code_label_recall_21: 0.6412 - cat_label_recall_19: 0.0000e+00 - cat_label_recall_20: 0.0000e+00 - cat_label_recall_21: 0.0000e+00 - val_loss: 0.0040 - val_code_label_loss: 0.0040 - val_cat_label_loss: 0.0000e+00 - val_code_label_recall_19: 0.3072 - val_code_label_recall_20: 0.4144 - val_code_label_recall_21: 0.6156 - val_cat_label_recall_19: 0.0000e+00 - val_cat_label_recall_20: 0.0000e+00 - val_cat_label_recall_21: 0.0000e+00\\n\",\n      \"Epoch 8/20\\n\",\n      \"70/70 - 11s - loss: 0.0038 - code_label_loss: 0.0038 - cat_label_loss: 0.0000e+00 - code_label_recall_19: 0.3330 - code_label_recall_20: 0.4455 - code_label_recall_21: 0.6494 - cat_label_recall_19: 0.0000e+00 - cat_label_recall_20: 0.0000e+00 - cat_label_recall_21: 0.0000e+00 - val_loss: 0.0040 - val_code_label_loss: 0.0040 - val_cat_label_loss: 0.0000e+00 - val_code_label_recall_19: 0.3084 - val_code_label_recall_20: 0.4160 - val_code_label_recall_21: 0.6147 - val_cat_label_recall_19: 0.0000e+00 - val_cat_label_recall_20: 0.0000e+00 - val_cat_label_recall_21: 0.0000e+00\\n\",\n      \"Epoch 9/20\\n\",\n      \"70/70 - 10s - loss: 0.0037 - code_label_loss: 0.0037 - cat_label_loss: 0.0000e+00 - code_label_recall_19: 0.3374 - code_label_recall_20: 0.4527 - code_label_recall_21: 0.6558 - cat_label_recall_19: 0.0000e+00 - cat_label_recall_20: 0.0000e+00 - cat_label_recall_21: 0.0000e+00 - val_loss: 0.0040 - val_code_label_loss: 0.0040 - val_cat_label_loss: 0.0000e+00 - val_code_label_recall_19: 0.3098 - val_code_label_recall_20: 0.4209 - val_code_label_recall_21: 0.6162 - val_cat_label_recall_19: 0.0000e+00 - val_cat_label_recall_20: 0.0000e+00 - val_cat_label_recall_21: 0.0000e+00\\n\",\n      \"Epoch 10/20\\n\",\n      \"70/70 - 11s - loss: 0.0037 - code_label_loss: 0.0037 - cat_label_loss: 0.0000e+00 - code_label_recall_19: 0.3431 - code_label_recall_20: 0.4590 - code_label_recall_21: 0.6630 - cat_label_recall_19: 0.0000e+00 - cat_label_recall_20: 0.0000e+00 - cat_label_recall_21: 0.0000e+00 - val_loss: 0.0040 - val_code_label_loss: 0.0040 - val_cat_label_loss: 0.0000e+00 - val_code_label_recall_19: 0.3124 - val_code_label_recall_20: 0.4209 - val_code_label_recall_21: 0.6173 - val_cat_label_recall_19: 0.0000e+00 - val_cat_label_recall_20: 0.0000e+00 - val_cat_label_recall_21: 0.0000e+00\\n\",\n      \"Epoch 11/20\\n\",\n      \"70/70 - 10s - loss: 0.0037 - code_label_loss: 0.0037 - cat_label_loss: 0.0000e+00 - code_label_recall_19: 0.3484 - code_label_recall_20: 0.4653 - code_label_recall_21: 0.6685 - cat_label_recall_19: 0.0000e+00 - cat_label_recall_20: 0.0000e+00 - cat_label_recall_21: 0.0000e+00 - val_loss: 0.0040 - val_code_label_loss: 0.0040 - val_cat_label_loss: 0.0000e+00 - val_code_label_recall_19: 0.3134 - val_code_label_recall_20: 0.4231 - val_code_label_recall_21: 0.6167 - val_cat_label_recall_19: 0.0000e+00 - val_cat_label_recall_20: 0.0000e+00 - val_cat_label_recall_21: 0.0000e+00\\n\",\n      \"Epoch 12/20\\n\",\n      \"70/70 - 10s - loss: 0.0036 - code_label_loss: 0.0036 - cat_label_loss: 0.0000e+00 - code_label_recall_19: 0.3527 - code_label_recall_20: 0.4717 - code_label_recall_21: 0.6731 - cat_label_recall_19: 0.0000e+00 - cat_label_recall_20: 0.0000e+00 - cat_label_recall_21: 0.0000e+00 - val_loss: 0.0040 - val_code_label_loss: 0.0040 - val_cat_label_loss: 0.0000e+00 - val_code_label_recall_19: 0.3145 - val_code_label_recall_20: 0.4235 - val_code_label_recall_21: 0.6174 - val_cat_label_recall_19: 0.0000e+00 - val_cat_label_recall_20: 0.0000e+00 - val_cat_label_recall_21: 0.0000e+00\\n\",\n      \"Epoch 13/20\\n\",\n      \"70/70 - 10s - loss: 0.0036 - code_label_loss: 0.0036 - cat_label_loss: 0.0000e+00 - code_label_recall_19: 0.3590 - code_label_recall_20: 0.4789 - code_label_recall_21: 0.6803 - cat_label_recall_19: 0.0000e+00 - cat_label_recall_20: 0.0000e+00 - cat_label_recall_21: 0.0000e+00 - val_loss: 0.0040 - val_code_label_loss: 0.0040 - val_cat_label_loss: 0.0000e+00 - val_code_label_recall_19: 0.3162 - val_code_label_recall_20: 0.4231 - val_code_label_recall_21: 0.6141 - val_cat_label_recall_19: 0.0000e+00 - val_cat_label_recall_20: 0.0000e+00 - val_cat_label_recall_21: 0.0000e+00\\n\",\n      \"Epoch 14/20\\n\",\n      \"70/70 - 10s - loss: 0.0036 - code_label_loss: 0.0036 - cat_label_loss: 0.0000e+00 - code_label_recall_19: 0.3621 - code_label_recall_20: 0.4848 - code_label_recall_21: 0.6868 - cat_label_recall_19: 0.0000e+00 - cat_label_recall_20: 0.0000e+00 - cat_label_recall_21: 0.0000e+00 - val_loss: 0.0040 - val_code_label_loss: 0.0040 - val_cat_label_loss: 0.0000e+00 - val_code_label_recall_19: 0.3169 - val_code_label_recall_20: 0.4230 - val_code_label_recall_21: 0.6156 - val_cat_label_recall_19: 0.0000e+00 - val_cat_label_recall_20: 0.0000e+00 - val_cat_label_recall_21: 0.0000e+00\\n\"\n     ]\n    },\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Epoch 15/20\\n\",\n      \"70/70 - 10s - loss: 0.0035 - code_label_loss: 0.0035 - cat_label_loss: 0.0000e+00 - code_label_recall_19: 0.3681 - code_label_recall_20: 0.4904 - code_label_recall_21: 0.6936 - cat_label_recall_19: 0.0000e+00 - cat_label_recall_20: 0.0000e+00 - cat_label_recall_21: 0.0000e+00 - val_loss: 0.0040 - val_code_label_loss: 0.0040 - val_cat_label_loss: 0.0000e+00 - val_code_label_recall_19: 0.3172 - val_code_label_recall_20: 0.4236 - val_code_label_recall_21: 0.6135 - val_cat_label_recall_19: 0.0000e+00 - val_cat_label_recall_20: 0.0000e+00 - val_cat_label_recall_21: 0.0000e+00\\n\",\n      \"Epoch 16/20\\n\",\n      \"70/70 - 10s - loss: 0.0035 - code_label_loss: 0.0035 - cat_label_loss: 0.0000e+00 - code_label_recall_19: 0.3730 - code_label_recall_20: 0.4972 - code_label_recall_21: 0.6986 - cat_label_recall_19: 0.0000e+00 - cat_label_recall_20: 0.0000e+00 - cat_label_recall_21: 0.0000e+00 - val_loss: 0.0040 - val_code_label_loss: 0.0040 - val_cat_label_loss: 0.0000e+00 - val_code_label_recall_19: 0.3181 - val_code_label_recall_20: 0.4237 - val_code_label_recall_21: 0.6152 - val_cat_label_recall_19: 0.0000e+00 - val_cat_label_recall_20: 0.0000e+00 - val_cat_label_recall_21: 0.0000e+00\\n\",\n      \"Epoch 17/20\\n\",\n      \"70/70 - 10s - loss: 0.0035 - code_label_loss: 0.0035 - cat_label_loss: 0.0000e+00 - code_label_recall_19: 0.3783 - code_label_recall_20: 0.5039 - code_label_recall_21: 0.7058 - cat_label_recall_19: 0.0000e+00 - cat_label_recall_20: 0.0000e+00 - cat_label_recall_21: 0.0000e+00 - val_loss: 0.0040 - val_code_label_loss: 0.0040 - val_cat_label_loss: 0.0000e+00 - val_code_label_recall_19: 0.3177 - val_code_label_recall_20: 0.4228 - val_code_label_recall_21: 0.6123 - val_cat_label_recall_19: 0.0000e+00 - val_cat_label_recall_20: 0.0000e+00 - val_cat_label_recall_21: 0.0000e+00\\n\",\n      \"Epoch 18/20\\n\",\n      \"70/70 - 11s - loss: 0.0034 - code_label_loss: 0.0034 - cat_label_loss: 0.0000e+00 - code_label_recall_19: 0.3842 - code_label_recall_20: 0.5093 - code_label_recall_21: 0.7107 - cat_label_recall_19: 0.0000e+00 - cat_label_recall_20: 0.0000e+00 - cat_label_recall_21: 0.0000e+00 - val_loss: 0.0040 - val_code_label_loss: 0.0040 - val_cat_label_loss: 0.0000e+00 - val_code_label_recall_19: 0.3184 - val_code_label_recall_20: 0.4234 - val_code_label_recall_21: 0.6122 - val_cat_label_recall_19: 0.0000e+00 - val_cat_label_recall_20: 0.0000e+00 - val_cat_label_recall_21: 0.0000e+00\\n\",\n      \"Epoch 19/20\\n\",\n      \"70/70 - 11s - loss: 0.0034 - code_label_loss: 0.0034 - cat_label_loss: 0.0000e+00 - code_label_recall_19: 0.3890 - code_label_recall_20: 0.5161 - code_label_recall_21: 0.7166 - cat_label_recall_19: 0.0000e+00 - cat_label_recall_20: 0.0000e+00 - cat_label_recall_21: 0.0000e+00 - val_loss: 0.0040 - val_code_label_loss: 0.0040 - val_cat_label_loss: 0.0000e+00 - val_code_label_recall_19: 0.3195 - val_code_label_recall_20: 0.4247 - val_code_label_recall_21: 0.6139 - val_cat_label_recall_19: 0.0000e+00 - val_cat_label_recall_20: 0.0000e+00 - val_cat_label_recall_21: 0.0000e+00\\n\",\n      \"Epoch 20/20\\n\",\n      \"70/70 - 11s - loss: 0.0034 - code_label_loss: 0.0034 - cat_label_loss: 0.0000e+00 - code_label_recall_19: 0.3939 - code_label_recall_20: 0.5220 - code_label_recall_21: 0.7228 - cat_label_recall_19: 0.0000e+00 - cat_label_recall_20: 0.0000e+00 - cat_label_recall_21: 0.0000e+00 - val_loss: 0.0041 - val_code_label_loss: 0.0041 - val_cat_label_loss: 0.0000e+00 - val_code_label_recall_19: 0.3185 - val_code_label_recall_20: 0.4246 - val_code_label_recall_21: 0.6130 - val_cat_label_recall_19: 0.0000e+00 - val_cat_label_recall_20: 0.0000e+00 - val_cat_label_recall_21: 0.0000e+00\\n\",\n      \"time: 3min 32s\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"finetune_history = model.fit(\\n\",\n    \"    train_generator,\\n\",\n    \"    epochs=EPOCHS,\\n\",\n    \"    validation_data=valid_generator,\\n\",\n    \"    verbose=2,\\n\",\n    \")\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"conventional-makeup\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"logical-opinion\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"median-beads\",\n   \"metadata\": {},\n   \"source\": [\n    \"# Cold Start\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 144,\n   \"id\": \"particular-corps\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 512 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"m = model(\\n\",\n    \"    patient_dim=PATIENT_DIM,\\n\",\n    \"    max_visit=MAX_VISIT,\\n\",\n    \"    max_code=MAX_CODE,\\n\",\n    \"    max_demo=MAX_DEMO,\\n\",\n    \"    code_vocab=len(code2int),\\n\",\n    \"    demo_vocab=len(demo2int),\\n\",\n    \"    util_vocab=4,\\n\",\n    \"    date_vocab=365,\\n\",\n    \"    cat_vocab=len(cat2int),\\n\",\n    \")\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"demonstrated-congress\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 146,\n   \"id\": \"frozen-borough\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Model: \\\"TransF\\\"\\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"Layer (type)                    Output Shape         Param #     Connected to                     \\n\",\n      \"==================================================================================================\\n\",\n      \"demo_feature (InputLayer)       [(None, 2)]          0                                            \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"demo_embedding (Embedding)      (None, 2, 100)       2400        demo_feature[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"code_feature (InputLayer)       [(None, 30, 10)]     0                                            \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"date_feature (InputLayer)       [(None, 30)]         0                                            \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"lambda_9 (Lambda)               (None, 100)          0           demo_embedding[0][0]             \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"code_embed (Embedding)          (None, 30, 10, 100)  4983600     code_feature[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"lambda_10 (Lambda)              (None, 30, 10, 1)    0           code_feature[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"lambda_11 (Lambda)              (None, 30)           0           date_feature[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"util_feature (InputLayer)       [(None, 30)]         0                                            \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.expand_dims_3 (TFOpLambda)   (None, 1, 100)       0           lambda_9[0][0]                   \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"multiply_9 (Multiply)           (None, 30, 10, 100)  0           code_embed[0][0]                 \\n\",\n      \"                                                                 lambda_10[0][0]                  \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"date_embedding (Embedding)      (None, 30, 100)      36500       date_feature[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.__operators__.getitem_13 (Sl (None, 30, 1)        0           lambda_11[0][0]                  \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"util_embedding (Embedding)      (None, 30, 100)      400         util_feature[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.__operators__.getitem_12 (Sl (None, 30, 1)        0           lambda_11[0][0]                  \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.math.reduce_sum_6 (TFOpLambd (None, 30, 100)      0           multiply_9[0][0]                 \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"multiply_11 (Multiply)          (None, 30, 100)      0           date_embedding[0][0]             \\n\",\n      \"                                                                 tf.__operators__.getitem_13[0][0]\\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"multiply_10 (Multiply)          (None, 30, 100)      0           util_embedding[0][0]             \\n\",\n      \"                                                                 tf.__operators__.getitem_12[0][0]\\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.math.reduce_sum_7 (TFOpLambd (None, 1)            0           tf.expand_dims_3[0][0]           \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"add_9 (Add)                     (None, 30, 100)      0           tf.math.reduce_sum_6[0][0]       \\n\",\n      \"                                                                 multiply_11[0][0]                \\n\",\n      \"                                                                 multiply_10[0][0]                \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.ones_like_3 (TFOpLambda)     (None, 1)            0           tf.math.reduce_sum_7[0][0]       \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.concat_15 (TFOpLambda)       (None, 31, 100)      0           tf.expand_dims_3[0][0]           \\n\",\n      \"                                                                 add_9[0][0]                      \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.concat_19 (TFOpLambda)       (None, 31)           0           tf.ones_like_3[0][0]             \\n\",\n      \"                                                                 lambda_11[0][0]                  \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.concat_16 (TFOpLambda)       (None, 31, 100)      0           tf.expand_dims_3[0][0]           \\n\",\n      \"                                                                 add_9[0][0]                      \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.concat_18 (TFOpLambda)       (None, 31)           0           tf.ones_like_3[0][0]             \\n\",\n      \"                                                                 lambda_11[0][0]                  \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.concat_17 (TFOpLambda)       (None, 31, 100)      0           tf.expand_dims_3[0][0]           \\n\",\n      \"                                                                 add_9[0][0]                      \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"multihead_attention-0 (multihea (None, 31, 100)      30200       tf.concat_15[0][0]               \\n\",\n      \"                                                                 tf.concat_19[0][0]               \\n\",\n      \"                                                                 tf.concat_16[0][0]               \\n\",\n      \"                                                                 tf.concat_18[0][0]               \\n\",\n      \"                                                                 tf.concat_17[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"ffn-0 (ffn)                     (None, 31, 100)      20200       multihead_attention-0[0][0]      \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.__operators__.getitem_14 (Sl (None, 1, 100)       0           ffn-0[0][0]                      \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.compat.v1.squeeze_3 (TFOpLam (None, 100)          0           tf.__operators__.getitem_14[0][0]\\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"patient_embedding (Dense)       (None, 100)          10100       tf.compat.v1.squeeze_3[0][0]     \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"code_label (Dense)              (None, 3357)         339057      patient_embedding[0][0]          \\n\",\n      \"==================================================================================================\\n\",\n      \"Total params: 5,422,457\\n\",\n      \"Trainable params: 5,422,457\\n\",\n      \"Non-trainable params: 0\\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"None\\n\",\n      \"time: 33.7 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"m.compile(optimizer=\\\"RMSprop\\\", loss=tf.keras.losses.BinaryCrossentropy(), metrics=[tf.keras.metrics.Recall(top_k=5), \\n\",\n    \"                                                                                       tf.keras.metrics.Recall(top_k=10), \\n\",\n    \"                                                                                       tf.keras.metrics.Recall(top_k=30)])\\n\",\n    \"print(m.summary())\\n\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 147,\n   \"id\": \"structural-space\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 325 µs\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"# opt = tf.keras.optimizers.Adam(learning_rate=0.01)\\n\",\n    \"# m.compile(optimizer=opt, loss=tf.keras.losses.BinaryCrossentropy(), metrics=[tf.keras.metrics.Recall(top_k=5), \\n\",\n    \"#                                                                                        tf.keras.metrics.Recall(top_k=10), \\n\",\n    \"#                                                                                        tf.keras.metrics.Recall(top_k=30)])\\n\",\n    \"# print(m.summary())\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 148,\n   \"id\": \"cleared-swimming\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Epoch 1/20\\n\",\n      \"70/70 - 11s - loss: 0.0524 - recall_41: 0.1668 - recall_42: 0.2301 - recall_43: 0.3743 - val_loss: 0.0047 - val_recall_41: 0.1927 - val_recall_42: 0.2697 - val_recall_43: 0.4610\\n\",\n      \"Epoch 2/20\\n\",\n      \"70/70 - 9s - loss: 0.0045 - recall_41: 0.1983 - recall_42: 0.2773 - recall_43: 0.4633 - val_loss: 0.0045 - val_recall_41: 0.1966 - val_recall_42: 0.2817 - val_recall_43: 0.4795\\n\",\n      \"Epoch 3/20\\n\",\n      \"70/70 - 9s - loss: 0.0044 - recall_41: 0.2138 - recall_42: 0.3049 - recall_43: 0.4900 - val_loss: 0.0044 - val_recall_41: 0.2262 - val_recall_42: 0.3118 - val_recall_43: 0.5048\\n\",\n      \"Epoch 4/20\\n\",\n      \"70/70 - 9s - loss: 0.0043 - recall_41: 0.2468 - recall_42: 0.3443 - recall_43: 0.5371 - val_loss: 0.0043 - val_recall_41: 0.2426 - val_recall_42: 0.3308 - val_recall_43: 0.5269\\n\",\n      \"Epoch 5/20\\n\",\n      \"70/70 - 9s - loss: 0.0041 - recall_41: 0.2795 - recall_42: 0.3847 - recall_43: 0.5798 - val_loss: 0.0043 - val_recall_41: 0.2471 - val_recall_42: 0.3388 - val_recall_43: 0.5346\\n\",\n      \"Epoch 6/20\\n\",\n      \"70/70 - 9s - loss: 0.0039 - recall_41: 0.3166 - recall_42: 0.4276 - recall_43: 0.6244 - val_loss: 0.0043 - val_recall_41: 0.2561 - val_recall_42: 0.3418 - val_recall_43: 0.5380\\n\",\n      \"Epoch 7/20\\n\",\n      \"70/70 - 9s - loss: 0.0037 - recall_41: 0.3552 - recall_42: 0.4698 - recall_43: 0.6668 - val_loss: 0.0044 - val_recall_41: 0.2526 - val_recall_42: 0.3348 - val_recall_43: 0.5261\\n\",\n      \"Epoch 8/20\\n\",\n      \"70/70 - 10s - loss: 0.0035 - recall_41: 0.3911 - recall_42: 0.5110 - recall_43: 0.7065 - val_loss: 0.0045 - val_recall_41: 0.2426 - val_recall_42: 0.3263 - val_recall_43: 0.5186\\n\",\n      \"Epoch 9/20\\n\",\n      \"70/70 - 10s - loss: 0.0033 - recall_41: 0.4259 - recall_42: 0.5546 - recall_43: 0.7375 - val_loss: 0.0046 - val_recall_41: 0.2367 - val_recall_42: 0.3212 - val_recall_43: 0.4980\\n\",\n      \"Epoch 10/20\\n\",\n      \"70/70 - 9s - loss: 0.0031 - recall_41: 0.4652 - recall_42: 0.5950 - recall_43: 0.7707 - val_loss: 0.0048 - val_recall_41: 0.2311 - val_recall_42: 0.3182 - val_recall_43: 0.5070\\n\",\n      \"Epoch 11/20\\n\",\n      \"70/70 - 9s - loss: 0.0029 - recall_41: 0.5005 - recall_42: 0.6295 - recall_43: 0.8009 - val_loss: 0.0049 - val_recall_41: 0.2246 - val_recall_42: 0.3073 - val_recall_43: 0.4877\\n\",\n      \"Epoch 12/20\\n\",\n      \"70/70 - 9s - loss: 0.0027 - recall_41: 0.5412 - recall_42: 0.6672 - recall_43: 0.8270 - val_loss: 0.0052 - val_recall_41: 0.2148 - val_recall_42: 0.3006 - val_recall_43: 0.4708\\n\",\n      \"Epoch 13/20\\n\",\n      \"70/70 - 9s - loss: 0.0025 - recall_41: 0.5758 - recall_42: 0.7015 - recall_43: 0.8495 - val_loss: 0.0054 - val_recall_41: 0.2174 - val_recall_42: 0.2999 - val_recall_43: 0.4682\\n\",\n      \"Epoch 14/20\\n\",\n      \"70/70 - 9s - loss: 0.0024 - recall_41: 0.6045 - recall_42: 0.7298 - recall_43: 0.8718 - val_loss: 0.0056 - val_recall_41: 0.1997 - val_recall_42: 0.2814 - val_recall_43: 0.4534\\n\",\n      \"Epoch 15/20\\n\",\n      \"70/70 - 9s - loss: 0.0022 - recall_41: 0.6374 - recall_42: 0.7570 - recall_43: 0.8873 - val_loss: 0.0058 - val_recall_41: 0.2096 - val_recall_42: 0.2888 - val_recall_43: 0.4469\\n\",\n      \"Epoch 16/20\\n\",\n      \"70/70 - 9s - loss: 0.0021 - recall_41: 0.6599 - recall_42: 0.7797 - recall_43: 0.9020 - val_loss: 0.0059 - val_recall_41: 0.2007 - val_recall_42: 0.2748 - val_recall_43: 0.4368\\n\",\n      \"Epoch 17/20\\n\",\n      \"70/70 - 9s - loss: 0.0019 - recall_41: 0.6871 - recall_42: 0.8025 - recall_43: 0.9161 - val_loss: 0.0063 - val_recall_41: 0.2057 - val_recall_42: 0.2788 - val_recall_43: 0.4330\\n\",\n      \"Epoch 18/20\\n\",\n      \"70/70 - 10s - loss: 0.0018 - recall_41: 0.7072 - recall_42: 0.8206 - recall_43: 0.9282 - val_loss: 0.0064 - val_recall_41: 0.1902 - val_recall_42: 0.2625 - val_recall_43: 0.4166\\n\",\n      \"Epoch 19/20\\n\",\n      \"70/70 - 9s - loss: 0.0017 - recall_41: 0.7264 - recall_42: 0.8364 - recall_43: 0.9356 - val_loss: 0.0066 - val_recall_41: 0.1876 - val_recall_42: 0.2670 - val_recall_43: 0.4220\\n\",\n      \"Epoch 20/20\\n\",\n      \"70/70 - 9s - loss: 0.0016 - recall_41: 0.7448 - recall_42: 0.8537 - recall_43: 0.9429 - val_loss: 0.0068 - val_recall_41: 0.1791 - val_recall_42: 0.2466 - val_recall_43: 0.4012\\n\",\n      \"time: 3min 7s\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"cold_start_his = m.fit(\\n\",\n    \"    train_generator,\\n\",\n    \"    epochs=EPOCHS,\\n\",\n    \"    validation_data=valid_generator,\\n\",\n    \"    verbose=2,\\n\",\n    \")\\n\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"quick-license\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"apparent-percentage\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"shaped-variable\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"japanese-canvas\",\n   \"metadata\": {},\n   \"source\": [\n    \"# plot\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 118,\n   \"id\": \"comprehensive-playback\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 5.8 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"import matplotlib.pyplot as plt\\n\",\n    \"\\n\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 150,\n   \"id\": \"established-probe\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 1.2 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"ft_recall_5 = [0.2107] + finetune_history.history['val_code_label_recall_19']\\n\",\n    \"ft_recall_10 = [0.3366] + finetune_history.history['val_code_label_recall_20']\\n\",\n    \"ft_recall_30 = [0.5424] + finetune_history.history['val_code_label_recall_21']\\n\",\n    \"\\n\",\n    \"cs_recall_5 = cold_start_his.history['val_recall_41']\\n\",\n    \"cs_recall_10 = cold_start_his.history['val_recall_42']\\n\",\n    \"cs_recall_30 = cold_start_his.history['val_recall_43']\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 151,\n   \"id\": \"gothic-evaluation\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"image/png\": \"iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA8Z0lEQVR4nO3deXxU1fn48c+TnX0N+yoCsoggERRFkEVxC1qxoFhxaa0Lrq1Wf1pLrbZWtN9qtSpuuFBRcUMFARXrAihBQVkkLGUJSwh7Qsj+/P64NzCGmcnMJJOZSZ736zWvzNx7z73PJJP7zLnn3HNEVTHGGGMqiot0AMYYY6KTJQhjjDFeWYIwxhjjlSUIY4wxXlmCMMYY41VCpAOoLi1bttQuXbpEOgxjjIkpy5Yt262qqd7W1ZoE0aVLFzIyMiIdhjHGxBQR2exrnV1iMsYY45UlCGOMMV5ZgjDGGOOVJQhjjDFehTVBiMgYEVkrIutF5G4v668XkR9FZLmIfCUivd3lo0VkmbtumYiMCGecxhhjjhW2XkwiEg88BYwGsoClIjJbVVd7bPYfVX3G3T4d+AcwBtgNXKiq20WkLzAPaB+uWI0xJhLSHlzA7ryiY5a3bJhExn2jIxDRz4WzBjEIWK+qG1W1CJgJjPXcQFUPerxsAKi7/HtV3e4uXwXUE5HkMMZqjDE1zlty8Le8poXzPoj2wFaP11nA4IobichNwB1AEuDtUtIlwHeqWhiOII0xsS+S38SDPbaqcrCghOyDBX73uz+/iCb1EhGRajluKCJ+o5yqPgU8JSKXA/cBk8rXiUgf4O/A2d7Kish1wHUAnTp1Cn+wxpiwqcoJryrfxKt6ovV37Be/+h/ZBwvYebCAnQcKyD5YQPbBQg4Xl1a63/4PLCAxXkhtmExqI89HCqmNkmuk9hHOBLEN6OjxuoO7zJeZwNPlL0SkA/AucKWqbvBWQFWnAdMA0tLSbOYjYyKspk7y+UUl7MkrYs+hIvbk+b+48MriTdRLjKd+UgL1k+JJSYynfpLzqJcU7/e4323Zx8HDxRwsKHF/FnPwcAkHjjwv9nvsBz5cTVJ8HK2bJNOmcQp92zdhZK8U2jROoXWTFG55/XufZf94QW9ycgudR14h2/YXsHzrfvYcKqKm5nkLZ4JYCnQXka44iWECcLnnBiLSXVXXuS/PB9a5y5sCHwF3q+rXYYzRGFNBuE7yqkpRaRmFJWUUFpdRWFL6s+f+XPXSt+w9VOQmhUIKissCfj/3v78q4G0r+sW/Fx2zLCk+jsb1EmlcL4HGKYl+y3/3x9E0q+/7MpG/BHHtGV29Li8pLWPvoSIG/fVTv8euDmFLEKpaIiKTcXogxQMvquoqEXkAyFDV2cBkERkFFAP7OHp5aTJwPHC/iNzvLjtbVXeFK15jjMPfSX7F1v1HvkWXf4P2fO1P13vmVCGmQlo0SOb41Ia0aJhE8wbJtGiQ5D5P4mIvJ/JyGfeN4nBRKflFpeQXlXC4uPTI68NFpdz19g8+y06/+hQnGaQcTQgpifE/26bL3R/5LN+8QZLf99WyYZLPZOxLQnwcrRqn+N1vdQlrG4SqzgHmVFh2v8fzW32UexB4MJyxGVNbhdJouuNAARty8li/K8/vvsc+dWyFPj5OaJySQON6/r9N3zLieJIT40lOiDv6MyGO5IR4khPjuPqlpT7LfnjzUL/79qdlQ/8dIP0liOE9W4V83EBEQ1dWfyLeSG2MqV7+agBrd+ayISePDbvyWJ+Tx4acPDbmHCK/qPJGU4Dnr0yjSf2ff6OunxR/5BKKv2/Td5zdM/g3E6BQvonH+rFr4riWIIyJQqG0A5SUlpFTSYPtOf/84sjz9k3r0a1VQ07p0pxuqQ2dR6sGDHrI97XtUb1bB/gOgleVE15VvolX9UQbqVpATRzXEoQxYRKuxt43M7aSfaCA7NwCdh4oPNKNcndeYaW9Wx6f0J9uqQ05LrUB9ZOq/98/Uif5qoj2yzyRZAnCmDCprNtmcWkZe/KK3G6MBUe7NOb6rwXcNcu5Zt60fqLTXbJxCr3bNqZ142RaN0nh3ndX+iw7tn/lI9bE4knehIclCGMiYOBfFrA333t/9iaVNPZ+cedZtGqcfExvmnL+EkQg7CRvylmCMKaalJSWsWr7Qb79316+3bTX77Zj+rY5emesx52yLRs6J35/jb2dWtT3u+9INtia2sUShDF++GtH+OoPI1i+dT9L3YTw3eZ9HHJ7A3Wp5CT+0MUnhiVesBqAqT6WIIzxw187Qr8p8ykqLUMEerZuxCUDOzCoa3MGdWlOq8YpfmsBlbFagIkGliBMrRZsT6JDhSVk7TtM1r58svYd9rvvq0/vwildmpPWpRlN6x974rbGXhPrLEGYWs1fDeCVxZt+lgy27s1nX77/4SI83XNeL7/r7SRvYp0lCBP1Aq0FlJYpu3IL2L6/gO37D7PjgP8awP3vryIpIY4OzerRoVl9+rZvQodm9ejYrP6RZac89Em1vx9jYoUlCBP1/NUCbnn9ezcZODeLlZYFPg7yt/eOpGWDZOLivI+0aUxdZwnCRC1VZV0lg8ct37qftk1SGNy1OW2bptCuaT3aNalHu6b1aNs0hX5T5vss26pR5SNiWmOxqcssQZiosjuvkK/X7+aLzN18tT6H7IP+7yr+4q6zwhqPtSOYuswShKkR/toR/jl+AF+uz+HLzN2s3nEQcIaROOP4lgzt3pI/vP1jyMe1GoAxobMEYWqEv3aEK174hsR4YWDnZtx5Tk+Gdm9Jn3ZNiHfbBqqSIKwGYEzoLEGYsKus4filq05hUNfmNEj2/nG0WoAxkWEJwoRFcWkZSzbuYe7KncxftdPvtmed4H/WLqsFGBMZliBMtSksKeWrdbuZu3Inn6zJZn9+MfWT4jmrZys++nFHpMMzxgTJEoQJmK+G5sYpCZx1Qis+W7OL3MISGqUkMKpXa8b0bcOwHqmkJMbzURXGJTLGREZYE4SIjAEeB+KB51X14QrrrwduAkqBPOA6VV3trrsHuNZdd4uqzgtnrKZyvhqaDxaU8EVmDued2JYxJ7bh9G4tSUqI+9k21o5gTOwJW4IQkXjgKWA0kAUsFZHZ5QnA9R9VfcbdPh34BzBGRHoDE4A+QDvgExHpoaqBzaxuqp1WMpfl0ntHkRAf53O9tSMYE3vCWYMYBKxX1Y0AIjITGAscSRCqetBj+wZA+VloLDBTVQuB/4nIend/i8MYr/Fif34R736/jde/3eJ3O3/JwRgTm8KZINoDWz1eZwGDK24kIjcBdwBJwAiPsksqlD1mMl0RuQ64DqBTp07VErRxagtLN+3j9W+38NGPOygqKeOkjk0jHZYxpoZFvJFaVZ8CnhKRy4H7gElBlJ0GTANIS0sLfJQ249XeQ0W8810Wr3+7hQ05h2iUnMD4tI5MGNSRPu2aVGkCHGNM7AlngtgGdPR43cFd5stM4OkQy5oA+eqJlJwQhyoUlZYxoFNTHhnXjwv6taV+0tGPiDU0G1O3hDNBLAW6i0hXnJP7BOByzw1EpLuqrnNfng+UP58N/EdE/oHTSN0d+DaMsdYZvnoiFZaUcdWQLkwY1JET2jT2uo01NBtTt4QtQahqiYhMBubhdHN9UVVXicgDQIaqzgYmi8gooBjYh3t5yd3uTZwG7RLgJuvBVHWVDXkxJb1PDUVijIkFYW2DUNU5wJwKy+73eH6rn7IPAQ+FL7q6Y+vefN7K2MqsZVmRDsUYE0Mi3khtwqOguJT5q7N5c+lWvt6wG4Ch3VPZfqAgwpEZY2KFJYhaZvX2g7yZsZV3v9/GgcPFtG9aj9tG9mBcWgfaN61nPZGMMQGzBBFjfPVCapgcz3GpDfkh6wBJ8XGc3ac1E07pxJBuLX4257L1RDLGBMoSRIzx1Qspr7CUopIy/nRhby7q355mDbyf8K0nkjEmUJYgapG5tw5FRCrf0BhjAmAJIkZs3ZvPK4s3+d3GkoMxpjpZgohiqsqiDXuYvmgTn6zJJt4SgDGmBlmCiEL5RSW8+/02Xl60iczsPFo0SGLyWcczcXBnTv3bp5EOzxhTR1iCiABfPZGa109kXFpHZn67hYMFJfRp15hHLz2JC/q1JSUxHrBeSMaYmmMJIgJ89UTam1/MC1/9jzF923D1kC4M7NzsmHYF64VkjKkpliCizFd/OIu2TepFOgxjjMGmAYsylhyMMdHCEoQxxhivLEHUsKKSskiHYIwxAbEEUcOmfLDK5zrriWSMiSbWSF2DXl2ymf98s4Xrh3Xj7nNPiHQ4xhjjl9UgasiSjXv48+xVjDihFXee0zPS4RhjTKUsQdSArXvzuXHGd3RuUZ9/TuhPfJwNmWGMiX6WIMLsUGEJv3klg+LSMp67Mo3GKYmRDskYYwIS1gQhImNEZK2IrBeRu72sv0NEVovIDyLyqYh09lj3iIisEpE1IvKExOBQpWVlyu/fWkFmdi5PXn4yx6U2jHRIxhgTsLAlCBGJB54CzgV6A5eJSO8Km30PpKlqP2AW8IhbdghwOtAP6AucAgwLV6zh8q/P1jN35U7uObcXw3qkRjocY4wJSjhrEIOA9aq6UVWLgJnAWM8NVHWhqua7L5cAHcpXASlAEpAMJALZYYy12n28cif/90kmvxjQnl8P7RrpcIwxJmjhTBDtga0er7PcZb5cC8wFUNXFwEJgh/uYp6prKhYQketEJENEMnJycqot8Kr6aedB7nhzOSd1bMpff3GiTeRjjIlJUdFILSJXAGnAVPf18UAvnBpFe2CEiAytWE5Vp6lqmqqmpaZGxyWcvYeK+M0rGTRMTmDarwYeGabbGGNiTTgTxDago8frDu6ynxGRUcC9QLqqFrqLLwaWqGqequbh1CxOC2Os1aK4tIybZnxH9sFCnv3VQFo3Tol0SMYYE7JwJoilQHcR6SoiScAEYLbnBiIyAHgWJzns8li1BRgmIgkikojTQH3MJaZo85cPV7N44x7+dvGJDOjULNLhGGNMlYQtQahqCTAZmIdzcn9TVVeJyAMiku5uNhVoCLwlIstFpDyBzAI2AD8CK4AVqvpBuGKtDq9/u4VXFm/m12d05ZKBHSovYIwxUU5UNdIxVIu0tDTNyMioueP5mDa0ZcMkm/XNGBMzRGSZqqZ5WxcVjdSxyNe0ob6WG2NMrLEEYYwxxitLEMYYY7yyBGGMMcYrSxDGGGO8sgQRIl/Tg9q0ocaY2sKmHA1Rxn2juf7VZWRm5/LZ74dHOhxjjKl2VoOogsxduXRvbXM8GGNqJ0sQISooLmXT7kP0bN0o0qEYY0xYWIII0cacQ5QpdLcEYYyppSxBhCgzOxeAnm0sQRhjaidLECHKzM4lIU7o0qJBpEMxxpiwsAQRoszsPLq2bEBSgv0KjTG1k53dQpSZnUsPu7xkjKnFLEGEIL+ohK378unRyhKEMab2sgQRgvW78lCFnm3sHghjTO1lCSIEmdl5gHVxNcbUbkEnCHeO6V+IyAnhCCgWZGbnkpQQR+fm9SMdijHGhE2lCUJE3vN4Phb4DLgQeF9ErgpbZFEsMzuXbqkNSYi3CpgxpvYK5AzX2eP5H4ARqno1cDpwu7+CIjJGRNaKyHoRudvL+jtEZLWI/CAin4pIZ491nURkvoiscbfpEthbCr/Mnbn0tDGYjDG1XCAJQj2eJ6jq/wBUdTdQ5quQiMQDTwHnAr2By0Skd4XNvgfSVLUfMAt4xGPdK8BUVe0FDAJ2BRBr2OUWFLP9QIG1Pxhjar1Ahvs+SUQOAgIki0hbVd0hIklAvJ9yg4D1qroRQERmAmOB1eUbqOpCj+2XAFe42/bGSUYL3O3ygnhPYVXeQG2D9BkTvOLiYrKysigoKIh0KHVOSkoKHTp0IDExMeAylSYIVfWVBOoDv/VTtD2w1eN1FjDYz/bXAnPd5z2A/SLyDtAV+AS4W1VLPQuIyHXAdQCdOnXys+vqs84dg6mHJQhjgpaVlUWjRo3o0qULIhLpcOoMVWXPnj1kZWXRtWvXgMuF0ouphYjEqep+VV0cbHkf+7wCSAOmuosSgKHA74FTgOOAqyqWU9VpqpqmqmmpqanVEUql1mbnUi8xng7N6tXI8YypTQoKCmjRooUlhxomIrRo0SLomltAM8qJSDPgL8CJwA6gmYhsA25W1UM+im0DOnq87uAuq7jvUcC9wDBVLXQXZwHLPS5PvQecCrwQSLzhtC47j+6tGxIXZx/woEztDoe8NCM1aAV3rqv5eEzEWHKIjFB+74F0c20KzAHeVtVhqjpBVc8BXgUeFpGhItLcS9GlQHf3vokkYAIwu8K+BwDPAumquqtC2aYiUl4tGIFH20Ukrc3OtctLofCWHPwtNyZM4uPj6d+//5HHpk2bGDJkSMj7mz59Otu3b6/GCKNHIDWIPwKPqupCEXkV55v8bqAl8CNO4/V9wB2ehVS1REQmA/NwGrNfVNVVIvIAkKGqs3EuKTUE3nKz2xZVTVfVUhH5PfCpOCuWAc9Vw/utkn2HisjJLaSHdXE1JuzSHlzA7ryiY5a3bJhExn2jQ95vvXr1WL58+c+WLVq0KOT9TZ8+nb59+9KuXbuQ9xGtAmmDOFNV33afFwKXqeppwHhgD/AVcJa3gqo6R1V7qGo3VX3IXXa/mxxQ1VGq2lpV+7uPdI+yC1S1n6qeqKpXqeqxn5QalmkN1OGhWvk2ps7xlhz8La+Khg2dL32ff/45w4cPZ9y4cZxwwglMnDgRdT+fy5YtY9iwYQwcOJBzzjmHHTt2MGvWLDIyMpg4cSL9+/fn8OHDdOnShd27dwOQkZHB8OHDAZgyZQrXXHMNw4cP57jjjuOJJ544cvzXXnuNQYMG0b9/f377299SWlpKNAikBpEiIqLOb+lkYIW7fCVwsqqW1ZVripm7nC6uliCCVFkC+Gc/6HUh9E6HDoMgzu5Qrwv+/MEqVm8/GFLZ8c967x/Tu11j/nRhH79lDx8+TP/+/QHo2rUr77777s/Wf//996xatYp27dpx+umn8/XXXzN48GBuvvlm3n//fVJTU3njjTe49957efHFF3nyySd59NFHSUtLqzTun376iYULF5Kbm0vPnj254YYbWL9+PW+88QZff/01iYmJ3HjjjcyYMYMrr7wysF9GGAWSIL4FRuJ0Nf03MF9EFgOnAc+KyCnAqvCFGD3WZefSKDmBtk1SIh1K7CgphA9u9b9N696w9DlY8hQ0bAO9LoBe6dD5dIhPsAZuU628XWLyNGjQIDp06ABwpI2iadOmrFy5ktGjnUtbpaWltG3bNuhjn3/++SQnJ5OcnEyrVq3Izs7m008/ZdmyZZxyyimAk8BatWoV/BsLg0ASxEPAmyJyvqo+7/YoOg74B84lqtnApPCFGD3W7syle+uG1gsjUId2w8yJsHUJJDaAYi8d3hq0gsvfgIKDsG4+rH4fvp8BS5+H+i2g53nWwF1LVfZNv8vdH/lc98ZvT6vucI5ITk4+8jw+Pp6SkhJUlT59+rB4ceU9+xMSEigrcwaZqNit1Ne+J02axN/+9rdqegfVJ5Ab5TaKyE3AbBGZj3PHcylwnvv4naquDW+YkaeqZGbnMqZvm0iHEhuyV8Pr4yFvF4x7Efpe4n/7lMZw4jjnUZQP6z+BNbNh1Xs1Eq4x/vTs2ZOcnBwWL17MaaedRnFxMZmZmfTp04dGjRqRm5t7ZNsuXbqwbNkyzj33XN5++20/e3WMHDmSsWPHcvvtt9OqVSv27t1Lbm4unTt3rrRsuAV0sVdVv8G5pPQF0Avoi5Mohqjql+ELL3rszitiX34x3W0WucplzocXzoaSIrh6TuXJoaKk+k57xCXPw10bwhOjiXotGyYFtTyckpKSmDVrFn/4wx846aST6N+//5GeT1dddRXXX3/9kUbqP/3pT9x6662kpaURH+9vNCJH7969efDBBzn77LPp168fo0ePZseOHeF+SwERrSU9SNLS0jQjIyNs+1+0fjeXP/8NM349mNOPbxm248Q0VVjyb5h/H7TuC5fNhCbtq77fKU38rDtQ9f2bGrNmzRp69eoV6TDqLG+/fxFZpqpeW9grvcQkIrk4I7oKPx/ZVQBV1cahhxs71rpdXLvbPRDelRTBnN/Ddy87PZIufhaSGoT/uGVl1uvJmDCp9D9LVRupamOPn409X9dEkNEgMzuPpvUTSW2YXPnGdU3+XnjtF05yGPo7uPSV6k0ODfz06Hj/Rigtqb5jGWOOCKQG4W0YjSNUdW/1hRO9Mt0hNqwHUwU5mfCfX8LBbXDxNDhpfPUfw1tXVlX44lFY+CAUHIBxL0GidT82pjoF0s11GUcvMVWkOF1ea7XyHkxj+9e+W+mD4ut+BASunQ8dB9VcLCIw7E5IaQJz74QZ4+Cy1yHZOhEYU10C6eYa+ODhtdTOgwXkFpTYJEE+7zvQmk0OngZfB/WawrvXw8vpcMXbUN9vpdcYE6CAhvsu5w773R04UpdX1S+qO6hoUz6LnE0zGqX6/dKpObx1Fbx0LvzqXWhcx2t7xlSDgLt/iMivce6DmAf82f05JTxhRZfMnXV8kD5V2LCw8u0iqee5Tu3hwDZ44RzYY/dPmNA9/PDDzJgxI6Btp0+fTmpqKv3796d3794891xwA0+/9957rF4d/GwGs2fP5uGHHw66XDCCqUHcijO72xJVPUtETgD+Gp6woktmdi4tGybTvEHN36ATUQUHYPnrzrAXe2JgzKMuZ8Ck2fDaJfDiGKcm0aZvpKMyoYrgGFzz5s3jzTffDHj78ePH8+STT7Jr1y769OlDeno6rVu3PrK+pKSEhATvp9v33nuPCy64gN69ex+zzl+59PR00tPTva6rLsF0IC9Q1QIAEUlW1Z+AnuEJK7pkZufSs00duv8hexV8cBs81gs+/oPTEHzxs5GOKjDtT4ZrPob4RJh+Hmz5JtIRmVCFYQyuqVOnHhlm+/bbb2fEiBEAfPbZZ0ycOBGAgwcPUlRURGpqKps2bWLEiBH069ePkSNHsmXLFr/7b9WqFd26dWPz5s1H7rAePHgwd911Fxs2bGDMmDEMHDiQoUOH8tNPP7Fo0SJmz57NnXfeSf/+/dmwYQPDhw/ntttuIy0tjccff5wPPviAwYMHM2DAAEaNGkV2djbg1FwmT54MOHdz33LLLQwZMoTjjjuOWbNmhfw78hRMDSLLnV3uPWCBiOwDNldLFFGsrExZtyuPX6Z1rHzjWFZaDGs+cGoLm7+GhBToOw4G/RraDXC2mf9H39/ooklqTydJvHIRvHoRjH8Vjh8V6ahMRXPvhp0/hlb2pfO9L29zIpzr+7LL0KFDeeyxx7jlllvIyMigsLCQ4uJivvzyS84880wAPvnkE0aOHAnAzTffzKRJk5g0aRIvvvgit9xyC++9957P/W/cuJGNGzdy/PHHA5CVlcWiRYuIj49n5MiRPPPMM3Tv3p1vvvmGG2+8kc8++4z09HQuuOACxo0bd2Q/RUVFlI8MsW/fPpYsWYKI8Pzzz/PII4/w2GOPHXPsHTt28NVXX/HTTz+Rnp7+s/2FKuAEoaoXu0+niMhCoAnwcZUjiHLb9h8mv6iUnm1qSfuDz66qcUAZNO0Mox+AAb86tjdQLA2t3bSTkyT+0cu55FSRDRVeJw0cOJBly5Zx8OBBkpOTOfnkk8nIyODLL788UrP4+OOPufrqqwFYvHgx77zzDgC/+tWvuOuuu7zu94033uCrr74iOTmZZ599lubNnf+dSy+9lPj4ePLy8li0aBGXXnrpkTKFhYU+4xw//uj9RFlZWYwfP54dO3ZQVFRE167eO5ZedNFFxMXF0bt37yO1jKoKOEGIyKnAKlXNVdX/ikhjYABQq+vwR2eRqyWXmHxWz8vg8jedb9pxlQ8wFhMatoIyH3dZ21Dhkefnmz7gfwyuq30PBe5PYmIiXbt2Zfr06QwZMoR+/fqxcOFC1q9ff2SMom+//Zann346qP2Wt0FU1KCBM6JAWVkZTZs29TsPhbdy4NRi7rjjDtLT0/n888+ZMmWK1zKeQ4lX1xh7wbRBPA3kebzOc5fVanWqi2uPc2pPcjDGh6FDh/Loo49y5plnMnToUJ555hkGDBiAiLBq1SpOOOGEI6OwDhkyhJkzZwIwY8YMhg4dGtIxGzduTNeuXXnrrbcA5wS+YoUzOWfF4cIrOnDgAO3bO4NevvzyyyEdP1TBJIjyaUcBUNUyKqmBiMgYEVkrIutF5G4v6+8QkdUi8oOIfCoinSusbywiWSJybGquIZnZubRtkkLjlMRIhWDC5cC2SEdg/PHVtlXFNq+hQ4eyY8cOTjvtNFq3bk1KSsqRE//cuXMZM2bMkW3/9a9/8dJLL9GvXz9effVVHn/88ZCPO2PGDF544QVOOukk+vTpw/vvvw/AhAkTmDp1KgMGDGDDhmO7Z0+ZMoVLL72UgQMH0rJlzY4kHfBw3yLyDvA5R2sNNwJnqepFPraPBzKB0UAWsBS4TFVXe2xzFvCNquaLyA3AcFUd77H+cSAV2Kuqk/3FF67hvs9/4ktaNkzm5WsidKdwdatrQ2f7e78JKTD4ejjjdudubBN20T7c9+jRo3nllVdCmk40FgQ73HcwNYjrgSHANpwT/mDgOj/bDwLWq+pGVS0CZgJjPTdQ1YWqmu++XAJ08Ah6INAamB9EjNWqtExZvyuv9rQ/rJ0b6QiiS++x8PXj8ER/WPyUM3+2qdMWLFhQa5NDKAJOEKq6S1UnqGorVW2tqperqr+WvvbAVo/XWe4yX64F5gKISBzwGPB7fzGJyHUikiEiGTk5OYG9kSBs2ZtPYUlZ7biD+vB+594G8dHGEG1dVauLv8sUv5gGv/3C6cY77//Bk2nww1vOHBPGmKB6MfXAubzUWlX7ikg/IF1VH6xqECJyBZAGDHMX3QjMUdUsf8Nrq+o0YBo4l5iqGkdFa2vTEBvz7oVDOfCbT4/e11AXVNaVtW0/547rDZ/BgvvhnV/D4n/BqD9Dt7NqJkZjolQwN8o9B9wJPAugqj+IyH8AXwliG+B5d1kHd9nPiMgo4F5gmKqW1/FPA4aKyI1AQyBJRPJU9ZiG7nBaV1tmkVv3CSx/Dc64o24lh2B0GwFdh8PKWfDpX5wb7LqNhO3fw2EvU57YfRQhU1WbVyUCQun6GkyCqK+q31b4w/qbymsp0F1EuuIkhgnA5Z4biMgAnIQzxvNylapO9NjmKiCtppMDONOMdmxej/pJQQ16G10KDsIHt0DLnjDsD5GOJrrFxTkjw/ZKd+4o/2IqFOz3vq3dRxGSlJQU9uzZQ4sWLSxJ1CBVZc+ePaSkBDepVjBnvt0i0g13XmoRGQfs8BNQiYhMxhn1NR54UVVXicgDQIaqzgam4tQQ3nI/LFtUNbyjTwVhXXZe7M8BseB+yN0B18y3GdcClZgCQybDgInw9y6RjqZW6dChA1lZWYSjzdD4l5KSQocOHSrf0EMwCeImnOv9J4jINuB/wER/BVR1DjCnwrL7PZ5XOkCOqk4HpgcRZ7UoLi1j4+48RvSK4cbbjf+FZS/BaZOh4ymRjib21GsW6QhqnfI7mU1sCKYX00b3hJ4KnIDToHxGuAKLtE27D1FcqrFbgyjMg9mToXk3GHFfpKOpnT57yOkdZkwtVWmCcO9mvkdEnhSR0UA+MAlYD/wy3AFGytpYb6D+9AHYvxXGPgWJ9SIdTe30xSPweD/471Qo9D1UgjGxKpAaxKs48z78CPwGWAhcClysqmP9FYxlmdl5xAl0S43BBLF5EXz7LAy6DjqfFuloYpu/+yh++yV0GgILH4R/9nNuuivK9769MTEokDaI41T1RAAReR6nYbpT+eRBtVXmzly6tGhASmKMDV5XlA/v3+QM2z3qT5GOJvZV1pX18pmQtQwWPuR0CFj0JAy9AwZebZ0CTMwLJEEUlz9R1VIRyartyQEgc1dubF5eWvgQ7N0IV86GpAaVb2+qrsNA+NU7sHmx8/v/+G74+gnnslORl0tPdg+FiRGBXGI6SUQOuo9coF/5cxE5GO4AI6GguJRNuw/FXgP11qWw5N/Ot9fjhlW+valenU+Dqz50knPTjt6TA9g9FCZmVFqDUNUYu8ZSdRtzDlGmMTYHRHEBvH8jNGrnzAhnIue4YdD1TPhz00hHYkyVxPAtwuGzbpfzzS+mphn9799hdyZc8TakNI50NMbuEja1QDDDfdcZa3fmkhAndGkRI9fwt3/v9KAZcIUzZaiJfkufh7LSSEdhjF+WILzIzM7juNQGJCXEwK+npAjeu8mZf/nshyIdjQnUR7+D50fCtmWRjsQYn2LgDFjzMrNzY6f94cvHYNcquOCfNitatPF3D8UlL8DB7fDcSPjwdsj3MmKsMRFmbRAV5BeVsHVfPuMGBjeoVY2Z2t17L5jZN1vXyWhT2d+j+2hY+DfnpsbV78Pov8BJlzmjyhoTBeyTWMH6XXmoEr3TjPrqImldJ2NPShM492FnVrsWxzu90F46F3aujHRkxgBWgzhGZnYeUEtmkTOxoc2JcPXHsHyGczf2s2dCQhIUHz52W7vJztQgq0FUkJmdS1JCHJ1jpQeTqR3i4uDkX8HNy5yf3pIDWE3R1ChLEBVkZudyfGpD4uOsH7uJgPrN4cLHIx2FMYAliGNk7syN3vYHY4ypQZYgPOQWFLP9QAE9ovkOan9dJ40xphpZI7WHIw3UraI4QYx7AV6+0OlHf+K4SEdjImHHCmh7UqSjMHVAWGsQIjJGRNaKyHoRudvL+jtEZLWI/CAin4pIZ3d5fxFZLCKr3HXjwxlnuXXZMTAG04qZkNQITjg/0pGYcPJVI5Q4eOl82Ph5jYZj6qaw1SBEJB54ChgNZAFLRWS2qq722Ox7IE1V80XkBuARYDzOtKZXquo6EWkHLBOReaq6P1zxgjPNaL3EeNo3jdIpOovynRuq+lxk04jWdr66sh7cDq9dAq+Ng4ufsVqkCatw1iAGAetVdaOqFgEzgZ9NUaqqC1W1fI7GJUAHd3mmqq5zn28HdgGpYYwVgHXZefRo3ZC4aO3B9NNHUJTn3G1r6qbG7eDqudBxELx9LSz+d6QjMrVYOBNEe2Crx+ssd5kv1wJzKy4UkUFAErDBy7rrRCRDRDJycnKqGK5Tg4jqMZhWvA5NOjnzIJu6q15TuOId6HUhzLvHubmurCzSUZlaKCoaqUXkCiANGFZheVvgVWCSqh7zH6Cq04BpAGlpaVqVGPbnF5GTWxi9s8jl7oSNC+GMO2ysHuPMd33pyzDnTmeo99xsGPskxCdWfd++xvuyu7jrnHAmiG1AR4/XHdxlPyMio4B7gWGqWuixvDHwEXCvqi4JY5zA0R5MUTsP9Y9vgZbBSRMiHYmJFnHxcP5j0KgtLHwQDuXAL1+B5Cp+hm28L+MK51fRpUB3EekqIknABGC25wYiMgB4FkhX1V0ey5OAd4FXVHVWGGM8Yq3bgylqx2BaMRPap0HL7pGOxEQTERh2J1z4hFPDfPkCyAvxcqsq7NtUreGZ2Ba2GoSqlojIZGAeEA+8qKqrROQBIENVZwNTgYbAW+JM0bhFVdOBXwJnAi1E5Cp3l1ep6vJwxbsuO5dGyQm0bZISrkOEbuePkL0Szns00pGYaDVwkjNp1FtXwYtnO20Uzbv6L1NWBrvXwuavYfNi2LwIcrfXSLgmNoS1DUJV5wBzKiy73+O51/kxVfU14LVwxlbR2p25dG/dEInGuYRXzIS4ROjzi0hHYqJZz3Phytnw4jnwRP9j1zdoBZe/4SSCzYtgy2I47E5U1KgtdB4CnU6DOb+v0bBN9IqKRupIU1Uys3MZ07dNpEM5VmmJ0/7Q4xxo0CLS0Zho12kw4KO/xqFd8NxZzvPmx0HP85yk0Pk0aNbVuVwFliDMEZYggN15RezLL6Z7NA6xsfFzyMu2xmlTPca96HSTbtzW9zYNWvnuxWTqFEsQRPkQGz/MhJSm0P3sSEdiaoO+l1S+TcWurB/eDsumwxVvhyUkE72sQz1HezBFXRfXgoOw5kPnnzohOdLRmLpq5P1Qrzl89Du7Ia+OqdM1iLQHF7A7r+jI60EPfQpAy4ZJZNw3OlJhHbVmNpQctqE1TGTVawZn/wXeuwGWvwYnXxnpiEwNqdM1CM/kEMjyGrdiJjTvBh3SIh2JiSXhmDPkpMuctosFf4L8vaHvx8SUOl2DiGr7t8CmL+Gse4/2LjEmEOEYDkPEuWv7mTPgkymQ/kT1H8OG+Ig6dboGEdV+eNP52e+XkY3DmHKte8OpN8B3L8PWpdW/fxviI+pYgohGqs7lpU5DoFmXSEdjzFHD74ZG7eCj2517dEytZgkiGm3/Dvass3sfTPRJbgRj/uYM/7L0+UhHY8KsTieIlg2TglpeY1bMhPhkZ+Y4Y6JN77HQbSQsfMgZhr465NllpGhUpxupo6Ira0UlRfDjLGfO6ZQmkY7GmGOJwHlT4d+nwvz74JIq1iT2bYJXL/a/zU9z4ITzqnYcX6xx3Kc6XYOISus/cQZQs8tLJpq16AZn3O6ME7bxv6HvJ3s1vHCO03W2XjPv28QlwMzL4ct/OO1z1c0ax32q0zWIqLTidWiQCt1GRDoSY/w743b44Q3nDusbFkFCkJdmt3wD/7kUEuvDNR9Dq17etys+DO/fBJ/+GXatgfR/OTPqmbCzGkQ0ObwPMj+GEy+tnqkjjQmnxHrOHCV71sHifwVXdt0CeGUs1G8B18zznRzKj3PJCzDiPvjxTZh+XvW1fRi/LEFEk1XvQmkR9Bsf6UiMCUz30dDrQvjvVOfmzkD8OAten+DMjnjNPGjWufIyInDmnTD+Ndj1E0w7C7Z/X7XY92+Fj++p2j5qOUsQ0WTFTEjtBW1PinQkxgRuzMPOCXzu3ZVv+800ePvX0PFUuOpDZxa8YPS6EK6d58zH/eK5sDKEEWZ3roR3roPHT4JvpwVfvg6xBBEt9myArd84jdM2tIaJJU06wLA/wNqPYO1c79uowsK/wdw7nYmKrng79F56bU6E3yx0vkjNugY+e6jyUWZV4X9fwGuXwDOnO6MkD74eblnue4yq5Cgc/r+GWSN1tPjhDUCc9gdjYs2pNzodLObeBV2HQVL9o+vKypzlS5+D/hPhwicgvoqnnoapMGk2fHgHfPEIfP04lBYeu12DVnDu3531O5Y7HUBG/BFOufZor6mKXVnLyuDViyArw/ni1qJb1WKNYWGtQYjIGBFZKyLrReSY+qeI3CEiq0XkBxH5VEQ6e6ybJCLr3MekcMYZceVDaxw3DJq0j3Q0xgQvIckZzG//FvjqH0eXlxTBO79xksOQm2HsU1VPDkeOmQxjn4Rz/uo9OYDTVXXW1VCYCxf8E25bCWf+3neXWoC4OLjoaSfOd39bp4cUEQ1Hv2JAROKBTGA0kAUsBS5T1dUe25wFfKOq+SJyAzBcVceLSHMgA0jDmWB3GTBQVff5Ol5aWppmZGSE5b2E3ebF8NIYuPhZu//BxLYHW0NJwbHLkxrA/9sevuNO8XO56pevOjeexsUHt88fZ8Hb18JZ98GwO6sWXxQTkWWq6nVOgXDWIAYB61V1o6oWATOBsZ4bqOpCVc13Xy4BOrjPzwEWqOpeNyksAMaEMdbIWvE6JDaAEy6IdCTGVI235ABQdKhm4/DUOz345ABw4jjoOw7++zBs+67644oB4UwQ7YGtHq+z3GW+XAuUt3AFVFZErhORDBHJyMnJqWK4EVJcAKvec3pnJEfZlKfG1HXnPwoNWzu9noryK9++lomKXkwicgXO5aSpwZRT1WmqmqaqaampqeEJLtwy50LhAbu0ZEw0qtcMLvq3czPggvsjHU2NC2cvpm1AR4/XHdxlPyMio4B7gWGqWuhRdniFsp+HJcpI8DY42KsX2eBgxoSqQSvfA+5V1XHD4dSbYMlT0GMMdB9V9X3GiHAmiKVAdxHpinPCnwBc7rmBiAwAngXGqKrnX3ce8FcRKe9qcDZQe255tMHBjKle4f5iNfJ+2PCZMybUjYuhfvPwHi9KhO0Sk6qWAJNxTvZrgDdVdZWIPCAi6e5mU4GGwFsislxEZrtl9wJ/wUkyS4EH3GXGmGjm6xt7dXyTj6TEFPjFNMjfAx/cGp5RZaNQ2Lq51rSY6OZaVgabv4KXL/S9zZQDNRePMSY4X/0ffDIFLnoG+l8W6Wiqhb9urnYndU3IyYQfZsIPb8KBrZVvb4yJTkNugcz5MOdO6DwksIEGY1hU9GKqlQ7tcQYmm3YWPHWK880j9QRn2GJjTGyKi4eLn3Gev3cDlJVGNp4wsxpEqHxNU5jcBLqcAevmQVmJM7DY2Q85Yyw1au1s8/E94etxYYwJr2ad4bxHnASx+Ek4/dZIRxQ2liBC5avHUeEB2JYBp94A/SZAm77HbmNdWY2JbSdd5oxc++lfnNkf25wY6YjCwhJEONy+uvoGJDPGRB8RZ/C/rd84d1n/ZmGtnAbVzmLhYMnBmNqvQQsY+2+YcQk81NrL+ti/8dUaqY0xJlT+7qquBTe+WoIwxhjjlSWIUNXWO0aNMdVn3QIoiN2bX+1ieahi/NqiMaYGzBgHEget+0Ln06HzadBpiDNlKvjuLh8l7ReWIIwxJlyunA1bFsPmr2HZdPjmaWd5yx7Q6bSoH7jTEoQxxlSFv6HGjxvmPMCZn3vHCidZbF7kTBQW5SxBGGNMVQR6KSghCTqe4jzOuM0ZpuOB6B423BqpjTEmEkKZJ7uGWYIwxphotObDSEdgCcIYYyLGV7f4uAR4YyJ88ueIjhhrbRDGGBMpvtovigtg7l3w1T9g+/fONAENWtRsbFgNwhhjok9iCqQ/Aen/cno8TRvmJIoaZgnCGGOi1clXwjUfO89fOAe+e7VGDx/WBCEiY0RkrYisF5G7vaw/U0S+E5ESERlXYd0jIrJKRNaIyBMiIuGM1RhjolL7k+G6/zp3Yc+eDB/cCiWFNXLosCUIEYkHngLOBXoDl4lI7wqbbQGuAv5ToewQ4HSgH9AXOAUYFq5YjTEmqjVoAVe8A2fc7tyR/dK5cCAr7IcNZyP1IGC9qm4EEJGZwFhgdfkGqrrJXVdWoawCKUASIEAikB3GWI0xJrrFxcOoKdB+ILx7A/xfX5xTZQXVOI5TOC8xtQe2erzOcpdVSlUXAwuBHe5jnqquqfYIjTEm1vS6EH7zGV6TA1TrOE5R2UgtIscDvYAOOEllhIgM9bLddSKSISIZOTk5NR2mMcZERmqPGjlMOBPENqCjx+sO7rJAXAwsUdU8Vc0D5gKnVdxIVaepapqqpqWmplY5YGOMMUeFM0EsBbqLSFcRSQImALMDLLsFGCYiCSKSiNNAbZeYjDGmBoUtQahqCTAZmIdzcn9TVVeJyAMikg4gIqeISBZwKfCsiKxyi88CNgA/AiuAFar6QbhiNcYYc6ywDrWhqnOAORWW3e/xfCnOpaeK5UqB34YzNmOMiWn+5qGoJjYWkzHGxKIamJI0KnsxGWOMiTxLEMYYY7yyBGGMMcYrSxDGGGO8sgRhjDHGK1H1MZ5HjBGRHGBzFXbREthdh8pG8tixWDaSx7b3HBtlI3nsqpTtrKreh6JQVXs4STKjLpWN1bjt92XvOVrLxnLcvh52ickYY4xXliCMMcZ4ZQniqGl1rGwkjx2LZSN5bHvPsVE2kseuatxe1ZpGamOMMdXLahDGGGO8sgRhjDHGqzqfIERkjIisFZH1InJ3kGVfFJFdIrIyhON2FJGFIrJaRFaJyK1BlE0RkW9FZIVb9s8hHD9eRL4XkQ+DLLdJRH4UkeUikhHCcZuKyCwR+UlE1ojIMTMF+ijX0z1m+eOgiNwWxHFvd39XK0XkdRFJCaLsrW65VYEc09vnQkSai8gCEVnn/mwWRNlL3WOXiUhakMed6v6ufxCRd0WkaRBl/+KWWy4i80WkXTDH9lj3OxFREWkZxLGniMg2j7/3ecEcV0Rudt/3KhF5JIjjvuFxzE0isjyIsv1FZEn5/4aIDPJW1k/5k0Rksfv/9YGINPZSzut5I9DPV9DC0Xc2Vh5APM7ERMcBSTiTE/UOovyZwMnAyhCO3RY42X3eCMgM9NiAAA3d54nAN8CpQR7/DuA/wIdBltsEtKzC7/xl4Nfu8ySgaYh/t504N/gEsn174H9APff1m8BVAZbtC6wE6uMMj/8JcHywnwvgEeBu9/ndwN+DKNsL6Al8DqQFedyzgQT3+d+DPG5jj+e3AM8Ec2x3eUecScM2+/rc+Dj2FOD3Afx9vJU9y/07JbuvWwUTs8f6x4D7gzjufOBc9/l5wOdBxr0UGOY+vwb4i5dyXs8bgX6+gn3U9RrEIGC9qm5U1SJgJjA20MKq+gWwN5QDq+oOVf3OfZ6LM+te+wDLqjpzdYOTIBKBgHsbiEgH4Hzg+aCCriIRaYLzj/ECgKoWqer+EHY1EtigqsHcOZ8A1BORBJyT/fYAy/UCvlHVfHVmSfwv8At/BXx8LsbiJEfcnxcFWlZV16jq2soC9VF2vhs3wBK8TNDlp+xBj5cN8PMZ8/O/8H/AXSGWrZSPsjcAD6tqobuNl1l1/B9XRAT4JfB6EGUVKP/W3wQ/nzEf5XsAX7jPFwCXeCnn67wR0OcrWHU9QbQHtnq8ziLAk3R1EpEuwACcmkCgZeLd6u8uYIGqBlwW+CfOP21ZEGXKKTBfRJaJyHVBlu0K5AAvuZe3nheRBiHEMAEf/7jeqOo24FGcuc53AAdUdX6AxVcCQ0WkhYjUx/lm2DHIeAFaq+oO9/lOoHUI+6iqa4C5wRQQkYdEZCswEbi/su0rlB0LbFPVFcGU8zDZvcT1YpCXTHrg/M2+EZH/isgpIRx7KJCtqsHMynMbMNX9fT0K3BPkMVdx9AvqpVTyOatw3gjL56uuJ4iIE5GGwNvAbRW+sfmlqqWq2h/nG+EgEekb4PEuAHap6rJQ4gXOUNWTgXOBm0TkzCDKJuBUq59W1QHAIZzqcMBEJAlIB94KokwznH+8rkA7oIGIXBFIWVVdg3NpZj7wMbAcKA0mZi/7VIKo8VUHEbkXKAFmBFNOVe9V1Y5uuclBHK8+8P8IMql4eBroBvTHSeqPBVE2AWgOnArcCbzp1giCcRlBfAlx3QDc7v6+bsetKQfhGuBGEVmGc/moyNeG/s4b1fn5qusJYhs/z9Id3GU1QkQScf7IM1T1nVD24V6iWQiMCbDI6UC6iGzCuaQ2QkReC+J429yfu4B3cS7TBSoLyPKo7czCSRjBOBf4TlWzgygzCvifquaoajHwDjAk0MKq+oKqDlTVM4F9ONd9g5UtIm0B3J9eL3uEg4hcBVwATHRPHqGYgZdLHn50w0nIK9zPWgfgOxFpE0hhVc12vwSVAc8R/OfsHfdS7Lc4NWWvDeTeuJchfwG8EcQxASbhfLbA+QITTMyo6k+qeraqDsRJTht8xOftvBGWz1ddTxBLge4i0tX9ZjoBmF0TB3a/0bwArFHVfwRZNrW8N4qI1ANGAz8FUlZV71HVDqraBef9fqaqAX2bFpEGItKo/DlOA2jAPbhUdSewVUR6uotGAqsDLe8K5ZvdFuBUEanv/t5H4ly7DYiItHJ/dsI5cfwnyOOD87ma5D6fBLwfwj6CJiJjcC4npqtqfpBlu3u8HEuAnzEAVf1RVVupahf3s5aF07i6M8Bjt/V4eTFBfM6A93AaqhGRHjidIYIZ6XQU8JOqZgVRBpw2h2Hu8xFAUJNGe3zO4oD7gGe8bOPrvBGez1d1tHTH8gPnmnImTra+N8iyr+NUf4tx/gGuDaLsGTjVwB9wLlssB84LsGw/4Hu37Ep89LQIYD/DCaIXE05vrxXuY1Wwvy93H/2BDDf294BmQZRtAOwBmoRw3D/jnOBWAq/i9nAJsOyXOIlsBTAylM8F0AL4FOek8QnQPIiyF7vPC4FsYF4QZdfjtLOVf8a89kTyUfZt9/f1A/AB0D7U/wX89H7zcexXgR/dY88G2gZRNgl4zY39O2BEMDED04HrQ/gbnwEscz8n3wADgyx/K865KBN4GHekiwrlvJ43Av18BfuwoTaMMcZ4VdcvMRljjPHBEoQxxhivLEEYY4zxyhKEMcYYryxBGGOM8coShDE+iEip/HwE2bvd5Z+LMwLwChH5uvy+DhFJEpF/ijMy8DoRed8d96p8f21EZKaIbHCHKpkjIj1EpIscOxrpFBH5vfv8VHfYiOXijIA7pQZ/DaYOS4h0AMZEscPqDGfizURVzXDHo5qKM/zHX3GGSOipqqUicjXwjogMdsu8C7ysqhPAGd4ZZ8ycrcfu/mdeBn6pqitEJB5nZFdjws4ShDFV8wVwmzv20NVAV1UtBVDVl0TkGpy7ahUoVtUjd8eqO4idO+iaP61wbqrC3Xewd58bExJLEMb4Vk9+PmHM31S14vg8F+Lc8Xs8sEWPHXAxA+jjPvc3QGK3CsdqgzMiKDhDZq8Vkc9xBgx8WVULAn0TxoTKEoQxvvm7xDRDRA7jDCFxM1DVGbw2eB7Ls51BVR8QkRk4Y19djjMe1fAqHs+YSlmCMCY0E1X1yJSrIrIX6CQijdSZyKXcQKB8WtdxoR5MVTcAT4vIc0COiLRQ1T2h7s+YQFgvJmOqgaoewmlM/ofbkIyIXIkze91n7iPZc5IlEeknIkMr27eInO8xn0F3nPko9lfvOzDmWJYgjPGtXoVurg9Xsv09QAGQKSLrcGYFu1hdOKOyjnK7ua4C/oYz+1dlfoXTBrEcZ5TTieUN4caEk43maowxxiurQRhjjPHKEoQxxhivLEEYY4zxyhKEMcYYryxBGGOM8coShDHGGK8sQRhjjPHq/wNFIswiUzm/jwAAAABJRU5ErkJggg==\\n\",\n      \"text/plain\": [\n       \"<Figure size 432x288 with 1 Axes>\"\n      ]\n     },\n     \"metadata\": {\n      \"needs_background\": \"light\"\n     },\n     \"output_type\": \"display_data\"\n    },\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 295 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"value1 = ft_recall_5\\n\",\n    \"value2 = cs_recall_5\\n\",\n    \"\\n\",\n    \"length = len(value1)\\n\",\n    \"plt.plot(range(length), value1, \\\"-s\\\", label=\\\"Finetune\\\")\\n\",\n    \"plt.plot(range(1, length), value2, \\\"-s\\\", label=\\\"w/o Pretrain\\\")\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"plt.xticks(range(length))\\n\",\n    \"plt.ylabel('Recall@5')\\n\",\n    \"plt.xlabel(\\\"EPOCHS\\\")\\n\",\n    \"plt.legend()\\n\",\n    \"plt.show()\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 152,\n   \"id\": \"confirmed-exhibit\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"image/png\": \"iVBORw0KGgoAAAANSUhEUgAAAY4AAAEGCAYAAABy53LJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA7NElEQVR4nO3dd3wUZf7A8c83nRJAgSBVEEIXEaIURVGK2CKnoCAqlhMLqGc78fQ87uQ8+53tJ4oiFhQ5VOROFFBAUaQkCkpvggaB0Dup398fM+HWsLvZSbLZlO/79dpXZmfmmXkm2cx3nzLPI6qKMcYYE6qoSGfAGGNMxWKBwxhjjCcWOIwxxnhigcMYY4wnFjiMMcZ4EhPpDJSFevXqafPmzSOdDWOMqVDS09N3qmr9wuurROBo3rw5aWlpkc6GMcZUKCKy2d96q6oyxhjjiQUOY4wxnljgMMYY44kFDmOMMZ5Y4DDGGONJWHtVicgA4DkgGnhNVR8PsN8VwFTgDFVNE5F+wONAHJAN3K+qc9x95wENgSNu8v6qmhnO6zDGmLKUMnY2Ow9mH7e+Xs040h7uF7a0oQpb4BCRaOAloB+QASwRkemqurLQfonAXcAin9U7gUtV9VcR6QjMBBr7bB+mqta/1pgqoKQ3wrK4kZY2f/kNtr600oYqnCWOM4H1qroRQEQmA5cBKwvt9yjwBHB/wQpV/d5n+wqgmojEq2pWGPNrTLkXyZtopNKW9EZYkvRlfc0Hs3LJ3H806HFfmrueKBGiBKJEEPdnlEB0lARNW1rCGTgaA7/4vM8AuvnuICJdgKaq+omI3I9/VwDfFQoab4hIHvABMFb9TCoiIiOAEQDNmjUr/lUYU44U5yaoquTkKdl5+UHT5+crUUFuPGX5LTgvXzmcncuR7Lygx527JhNVJS8f8lVRVfLVWc5X59qDOZiVS424aET8X3e4rvnVrzawfX8WmQeyyNx/9NjPQ0VcL8BTM9cUuU+4RezJcRGJAp4Frg+yTwec0kh/n9XDVHWLW8X1AXAt8FbhtKr6KvAqQEpKis1WZSo0VWVbEd9Ez3t6Htm5+eTk5ZOdl09Obv6xgBGKU/40g+goITZaiIuOIi4milifn8EMn7CY6CjnW6/4fBuOEgkajAAuem4+R3PyOJydx5GcPI5k54Wc5xveWBLSfoF0/MtMoqOEWgkx1K4WS+1qsdRyf9auFhs07ZjpKziSncdhN89HcnKda3CvI5jHZqymWmw0DWrFk5SYQIdGtTivTRJJteJpUCueu99fFjDtmrEDUJ/gmK+K5kOeKvmqpIz9vFi/Cy/CGTi2AE193jdx1xVIBDoC89xofxIwXURS3QbyJsBHwHWquqEgkapucX8eEJF3carEjgscxoRTuKowFv+pL7/uO8K6zIOs336QtdsPOMuZBzmYlRv0uKc2ru3e6J0bf2x0FLExUT5BQHhsxuqA6e/u25rsvDwn2BQEIPdnTp6yPvNgwLR7D2cf900/X5W8fKWoSUYb1UkgITaa6nHRVIuNplpcjM9yNA9PWx4w7Ue39/QJUD7ByieAnf/MlwHT/+mituw7kuO+co8tZ+w5wr4jOUHz/eF3GVSPi6GaT15rxsdQr2Y81eOi2bzrcMC0P47pT834mIAlnWCBIz4mOmi+ykI4A8cSIFlEWuAEjCHA1QUbVXUfUK/gvdtb6j43aNQBPgFGq+o3PvvEAHVUdaeIxAKXAOEPr8YUEqwaYvFPu4udtuOYmRz2qa6oVzOe5KSaXN6lMckNEvlzkJvo80NPLzLfwQLHXX2Tg6b9ZPQnAbd9POrsoGmbB0n72vAzgqYNFjhOb3ZC0LRFGXFOy6Dbg+X7hzEXBE378dJfA25LTAhemqlXMy7gl4uilCRtqMIWOFQ1V0RG4fSIigYmqOoKEfkbkKaq04MkHwW0Ah4RkUfcdf2BQ8BMN2hE4wSN8eG6BmP82Xc4+DfRK1/5ttjHvjKlKckNapKclEhyUk1OqPHbf/ZggaOyKumNsCxupKWtJL29yqKnWFjbOFR1BjCj0LpHAuzb22d5LDA2wGG7llb+jAnV0Zw8vliVycdLtzBvzY6g+076fbeg24e9tijgtjGpHYKmjeRNNFJpS3ojLEn68v7NP1KkqF4HlUFKSorasOrGq7x85dsNu5i2dAszl2/jQFYuSYnxpJ7WiNe+/ilguk2PXxz0uMGqP4pKa0xZEpF0VU0pvL5KzMdhjD+BGqnrVIvl8i5N+M8Pv7LjQBaJ8TEM6HgSA09vTPdT6hIdJUEDhzGVnQUOU2UFaqTeeySHdxZupneb+gw8vTHnt00iIfa3PVmsCsNUZRY4TIUWarfY3Lx8th/IYuveI2zZe4Rf9wZ/JmLJQ32pXT1wz5fy3nhpTDhZ4DAVWrCuraPe/Y6t+47y694jbN9/lHwPzXnBgoYxVZ0FDhNRXh6kU1V2HsxmXeYB1mceZN32wA+kASzfso+GtavRs2U9GtVJoFGdas6rdgIN61Sj419mluq1GFNVWOAwERWsxPDN+p2s236Ate5T1OsyD7DH5xmKxITgH995959Xqnk1xjgscJhyq+B5h9rVYmndoCYDOjakdcHDcQ1qkpQYT4sHZxRxlMCskdqY4rHAYcrU0Zw8fsjYR9rm3aRv2hN033dv7kZyUiL1asYFHNOnJKyR2pjiscBhSixYO8XMP5xD2uY9pG/eQ9qm3fy4ZR85eU4rdcv6NYIet2fLekG3F5zDSg3GlC0LHKbEgrVTdHWHeI6LjqJTk9rceHYLUk4+ka4nn8CJNeKCPkUdCis1GFP2LHCYEsk8EPx5iAcvbEtK8xPo2Li23+GgrcRgTMVjgcN4cjQnj8U/7Wb+uh3MX7eT1dsOBN3/lnODD1ttJQZjKh4LHAYI3k7x1o3djgWKxZt2k52bT1x0FCnNT+CBAW154rPAczwYYyofCxwGCN5OcdHz8wFo3aAm13Y/mV7J9ejWoi7V4pyqJwscxlQtFjgM+UWMxfH04NM4u1U9Tqqd4He7tVMYU7VY4Kiijubk8e2GXcxetZ0vVm0Puu+grk2Cbrd2CmOqFgsclUhR4z7tPpTNnNWZzF65jfnrdnI4O4/qcdGc27o+ny7fFoEcG2MqIgsclUiwdorB4xaQvnkP+Qon1Urg8i6N6duuAd1PqUtCbHSJn6cwxlQdYQ0cIjIAeA6IBl5T1ccD7HcFMBU4Q1XT3HUPAjcBecCdqjrTyzHNbx3KymPU+cn0a9eAjo1rHTeEh7VTGGNCFbbAISLRwEtAPyADWCIi01V1ZaH9EoG7gEU+69oDQ4AOQCPgcxFp7W4u8pjmeDPu6hV0u7VTGGNCFRXGY58JrFfVjaqaDUwGLvOz36PAE4DvI8iXAZNVNUtVfwLWu8cL9ZhVyqKNuxj66sJIZ8MYU0WEM3A0Bn7xeZ/hrjtGRLoATVW1cAV7oLRFHtPn2CNEJE1E0nbs2FG8KyjnlmzazdXjF3LVqwtZlxl8UiNjjCkt4QwcQYlIFPAscG84jq+qr6pqiqqm1K9fPxyniJj0zbu55rVFDB73LWu3H+Dhi9sx/4/nBWyPsHYKY0xpCmfj+Bagqc/7Ju66AolAR2Ce21B7EjBdRFKLSBvsmJVa+uY9/Ovztcxft5O6NeJ46KJ2XNP95GNPcFs7hTGmLIQzcCwBkkWkBc7NfQhwdcFGVd0HHJtwQUTmAfepapqIHAHeFZFncRrHk4HFgAQ7ZkUX6DmMOtVi6dS0Dl+t3cGJNeJ48MK2XNvjZKrHWW9qY0zZC9udR1VzRWQUMBOn6+wEVV0hIn8D0lR1epC0K0RkCrASyAVGqmoegL9jhusaylqg5zD2Hsnhx4y9PDCgLdf1OJka8RYwjDGRI6rBxymqDFJSUjQtLS3S2ShSsIfwVvz1AgsYxpgyJSLpqppSeH3EGseNNxY0jDHlhQUOY4wxnljgKCcy9hyOdBaMMSYkFjjKgZ93HeaqVxYiAbbbcxjGmPLEKs4jbNPOQwwdv5DD2Xn8546z6di4dqSzZIwxQVngiKCNOw4ydPxCsnPzee/m7rRvVCvSWTLGmCJZ4IiQ9ZkHGDp+Efn5ynsjutP2JAsaxpiKwQJHBKzZdoBhry0EhMkjupPcIDHSWTLGmJBZ43gZW7V1P0PHLyRKLGgYYyomK3GUoeVb9nHN64uoFhvNuzd3p0W9GpHOkjHGeGYljjLyQ8Zerh6/kBpxMbw/oocFDWNMhWUljjLw/c97uG7CYmpXi+W9m7vT9MTqkc6SMcYUmwWOUhZoaPQogc/+cA6N61SLQK6MMab0WFVVKQs0NHq+YkHDGFMpWOAwxhjjiQUOY4wxnljgMMYY44kFDmOMMZ6ENXCIyAARWSMi60VktJ/tt4rIjyKyVES+FpH27vph7rqCV76IdHa3zXOPWbAtKZzX4FWgIdBtaHRjTGURtjnHRSQaWAv0AzKAJcBQVV3ps08tVd3vLqcCt6vqgELHORWYpqot3ffzgPtUNeRJxMtyznFVpcc/5tC5aR3GXdu1TM5pjDHhEIk5x88E1qvqRlXNBiYDl/nuUBA0XDUAf1FsqJu2Qvhxyz627T9Kv/YNIp0VY4wJi3A+ANgY+MXnfQbQrfBOIjISuAeIA873c5yrKBRwgDdEJA/4ABirfopNIjICGAHQrFmz4uS/WGat2E50lHB+23JVg2aMMaUm4o3jqvqSWw31APCw7zYR6QYcVtXlPquHqeqpQC/3dW2A476qqimqmlK/fv0w5f54s1Zu44zmJ3BCDWvTMMZUTuEMHFuApj7vm7jrApkMDCy0bgjwnu8KVd3i/jwAvItTJVYubNp5iLXbD9K//UmRzooxxoRNOAPHEiBZRFqISBxOEJjuu4OIJPu8vRhY57MtCrgSn/YNEYkRkXrucixwCeBbGomo2Su3A1j7hjGmUgtbG4eq5orIKGAmEA1MUNUVIvI3IE1VpwOjRKQvkAPsAYb7HOIc4BdV3eizLh6Y6QaNaOBzYHy4rsGr2Su3065hLRv91hhTqYV1dFxVnQHMKLTuEZ/lu4KknQd0L7TuEFAu+7juPJhF2ubd3HF+ctE7G2NMBRbxxvHKYs6qTPLVqqmMMZWfBY5SMmvldhrXqUaHRrUinRVjjAkrCxyl4HB2LvPX7aBf+waISKSzY4wxYWWBoxTMX7eTrNx8+ls1lTGmCrDAUQpmrdhO7WqxnNHixEhnxRhjws4CRwnl5uUzZ/V2zm+bRGy0/TqNMZWf3elKKG3zHvYczrFqKmNMlWGBo4RmrdhOXEwU57Quu/GwjDEmkixwlICqMnvVNs5uVY8a8WF9ltIYY8oNCxwlsHrbAX7ZfcSqqYwxVYoFjhKYvXI7ItCnnQUOY0zVYYGjBGat3EaXZidQPzE+0lkxxpgyY4GjmLbsPcLyLfutmsoYU+VY4Cimz23uDWNMFWWBo5hmrdxGq6SanFK/ZqSzYowxZarIwOHOuneLiHwmIj+4r09F5FZ3QqUqZ9/hHBZt3G2lDWNMlRTKwwdvA3uBMUCGu64Jzmx97wBXhSNj5dncNZnk5qu1bxhjqqRQAkdXVW1daF0GsFBE1oYhT+XerJXbSEqM57QmdSKdFWOMKXOhtHHsFpHBInJsXxGJEpGrcOYJr1KO5uTx5Zod9G3fgKgom3vDGFP1hBI4hgCDgO0istYtZWwDLne3BSQiA0RkjYisF5HRfrbfKiI/ishSEflaRNq765uLyBF3/VIRGeeTpqubZr2IPC9lPHPStxt2cSg7z6qpjDFVVpFVVaq6CbcdQ0Tquut2FZVORKKBl4B+OFVbS0Rkuqqu9NntXVUd5+6fCjwLDHC3bVDVzn4O/TJwM7AImOHu/2lR+Skts1Zup2Z8DD1a1i2rUxpjTLniqTuuqu7yDRoi0i/I7mcC61V1o6pmA5OBywodb7/P2xqABju/iDQEaqnqQlVV4C1goJdrKIn8fOXzVds5t0194mOiy+q0xhhTrpR0SNfXgWYBtjUGfvF5nwF0K7yTiIwE7gHigPN9NrUQke+B/cDDqjrfPWaGzz4Z7rrjiMgIYARAs2aBsujN0oy97DiQZdVUxoRBTk4OGRkZHD16NNJZqXISEhJo0qQJsbGhPWFRZOAQkemBNgElrq9R1ZeAl0TkauBhnG6+W4FmqrpLRLoC00Skg8fjvgq8CpCSkhK0JBOqWSu2ExMl9G6TVBqHM8b4yMjIIDExkebNm1PGTZdVmqqya9cuMjIyaNGiRUhpQilx9AKuAQ4WWi841VGBbAGa+rxv4q4LZDJO+wWqmgVkucvpIrIBaO2mb+LhmKVq1spt9GhZl9rVquRzj8aE1dGjRy1oRICIULduXXbs2BFymlACx0LgsKp+6eeEa4KkWwIki0gLnJv7EODqQumTVXWd+/ZiYJ27vj6wW1XzROQUIBnYqKq7RWS/iHTHaRy/DnghhGsosfWZB9m44xDX92xeFqczpkqyoBEZXn/vRTaOq+qFqjo3wLZzgqTLBUYBM4FVwBRVXSEif3N7UAGMEpEVIrIUp51juLv+HOAHd/1U4FZV3e1uux14DVgPbKCMelTNdgc17GtzbxhTaUVHR9O5c+djr02bNtGzZ89iH2/ixIn8+uuvpZjD8sFz47jbJXePquYXta+qzsDpMuu77hGf5bsCpPsA+CDAtjSgo5c8l4bZK7dxauPaNKpTraxPbYwpJGXsbHYezD5ufb2acaQ9HKyzZ3DVqlVj6dKlv1m3YMGCYh9v4sSJdOzYkUaNGhX7GOVRSN1xReQEEXlRRL7EeTbjUxGZICI1wpu98iFz/1G+/2Wv9aYKh6eSYUzt419PJUc6Z6Yc8xc0gq0viZo1nRGw582bR+/evRk0aBBt27Zl2LBhOE8FQHp6Oueeey5du3blggsuYOvWrUydOpW0tDSGDRtG586dOXLkCM2bN2fnzp0ApKWl0bt3bwDGjBnDjTfeSO/evTnllFN4/vnnj53/nXfe4cwzz6Rz587ccsst5OXllfo1ehVKr6o6OKWGP6nqKJ/15wGPi8gUYIVPVVKl8/mqTFShf4eTIp2VyudQprf1pkr4639WsPLX/UXv6MdVr3zrd337RrX4y6XBO2ceOXKEzp07A9CiRQs++uij32z//vvvWbFiBY0aNeKss87im2++oVu3btxxxx18/PHH1K9fn/fff5+HHnqICRMm8OKLL/L000+TkpJSZL5Xr17N3LlzOXDgAG3atOG2225j/fr1vP/++3zzzTfExsZy++23M2nSJK677rrQfhlhEkpV1Z+Bp1V1roi8DXQHdgL1gB9xelc9jNNGUSnNXrmNZidWp3UDm3vDmMrMX1WVrzPPPJMmTZyOnQVtIHXq1GH58uX06+dUkeXl5dGwYUPP57744ouJj48nPj6epKQktm/fzhdffEF6ejpnnHEG4AS2pKTIPw4QSuA4R1XvdZezgKGqmiYiXYDbgK+B58KVwUg7mJXLN+t3cV2Pk63HR2nIzYJtP0JGGmxJi3RuTDlVVMmg+ehPAm57/5YepZ2dY+Lj448tR0dHk5ubi6rSoUMHvv3Wf0nHV0xMDPn5TvNw4QcdAx17+PDh/OMf/yilKygdoQSOBBERd4iPLsAyd/1yoIuq5lfmG+pXa3eQnZdvkzYF81Sy/6qlGklw42dukEh3AsW2HyHPrYdOLKLBcNEr0GU4xCaUfp6NKSVt2rRhx44dfPvtt/To0YOcnBzWrl1Lhw4dSExM5MCBA8f2bd68Oenp6Vx44YV88IHf/j+/0adPHy677DLuvvtukpKS2L17NwcOHODkk08O5yUVKZTG8cVAH3f5/4BZIvIYTjfbV0TkDGBFmPIXcbNWbOPEGnF0PfmESGel/ArWTvFCF/hoBHz/DsRUg+63wZVvwz2r4N5VwY/76R+d9GkTILf0Gz1NxVWvZpyn9eEUFxfH1KlTeeCBBzjttNPo3LnzsZ5Y119/PbfeeuuxxvG//OUv3HXXXaSkpBAdXfR4d+3bt2fs2LH079+fTp060a9fP7Zu3RruSyqSFPQKCLiD8wDeFOBiVd0uIvWAU4CNOIFnOjBcVYM9DBhRKSkpmpYWerVIuLr6VVpjagfedulz0DgF6reFaD8F3GCllSvGw5y/Q8ZiqNMMzn0AOg3xfxxT4a1atYp27dpFOhtVlr/fv4ikq+pxLfuhDKu+0R2IcLqIzMJ5kjwPuMh93Vueg0ZxlGVXvwpv67Lg27teH3z7/euCb29xLqz/HOaMhY9Hwvxnofdo6HgFRNkIxcZEQkhf3VR1kYj0wKmyOs1dvRAY6z4hbqqaQ7tgzqOQPjG85xGB5H7Qqi+smQFzH4MPb4b5zzgBZMYfA5dYigpKxphiCbnM7z4pPtt9maoqL9dpc5g7FrIOQrdbYdHL4T+vCLS9GFpfCKs+hrn/gH9fH3h/ew7EmLAJ5QHAAzgTLAm/nWhJAFXVWmHKmylvfvoKPn0AMlc6VUgXPgFJ7WD5B4G/9Ze2qCjo8Dtol+qc98ObS/8cxpigQmnjSCyLjJhybO/PMOvPsHKa00h95dvQ7lKnFACRqRKKioZOV1rgMCYCQilxnBhse2UcaqRezbiAvaoqrUC9m2JrQMF4luc9BD3vgNgKMtCj6v+CmzGm1ITSxpHO/6qqClOcrrmVSpXschuoTSDnkFM11O9RqNPU/z7l1ZuXwsXPQv3Wkc6JqUQef/xxmjZtyrBhw4rcd+LEidx///00btyY7Oxs7r77bm6+OfRS8rRp02jdujXt27f3lMfp06ezcuVKRo8e7SldqEKpqgptLkFTeQ2eGOkcBFYjyX/Qi0uEbT/Ayz3h7D9Ar3srTknJFC3Y8z9hrjqdOXMmU6ZMCXn/q666ihdffJHMzEw6dOhAamoqDRr8bySK3NxcYmL834qnTZvGJZdc4jdwBEuXmppKamqq322lwdOTVCJyAs5sfMfGgFDVr0o7U8aELNhN4uAOmPUQfPUU/DgVLn4GWvUJvL+pOMIwqvJTTz1FfHw8d955J3fffTfLli1jzpw5zJkzh9dff51Jkyaxf/9+srOzqV+/Pps2beLGG29k586d1K9fnzfeeINmzZoFPH5SUhItW7Zk8+bNPPDAAyQkJPD9999z1llnMXLkSEaOHMmOHTuoXr0648ePZ/fu3UyfPp0vv/ySsWPH8sEHH3DTTTfRuXNnvv76a4YOHUrr1q0ZO3Ys2dnZ1K1bl0mTJtGgQQMmTpxIWloaL774Itdffz21atUiLS2Nbdu28eSTTzJo0KBi/57AQ+AQkd8Dd+HM870UZ5Tcb4HzS5QDY8KlZn24/FXoPAw+uQfeudx5cPCCxyDRhsgv1z4d7YxrVhxvXOx//UmnwoWPB0zWq1cvnnnmGe68807S0tLIysoiJyeH+fPnc845zmSnn3/+OX36OF8+7rjjDoYPH87w4cOZMGECd955J9OmTQt4/I0bN7Jx40ZatWoFQEZGBgsWLCA6Opo+ffowbtw4kpOTWbRoEbfffjtz5swhNTWVSy655Dc3+uzsbApGwtizZw8LFy5ERHjttdd48skneeaZZ44799atW/n6669ZvXo1qampZRc4cILGGcBCVT1PRNoCj5Xo7MaUhVPOhdsWwNf/ch4cXDcb+jwCKTfa0+fmmK5du5Kens7+/fuJj4+nS5cupKWlMX/+/GMTK3322WfccMMNAHz77bd8+OGHAFx77bX88Y9/9Hvc999/n6+//pr4+HheeeUVTjzR6W80ePBgoqOjOXjwIAsWLGDw4MHH0mRlZQXM51VXXXVsOSMjg6uuuoqtW7eSnZ1Nixb+WxYGDhxIVFQU7du3Z/v27R5+K/55CRxHVfWoiCAi8aq6WkTalDgHJvIy0gNvC8ezGJEQEw+9H4BTBzmljxn3wWejId/PwAf21HnkBSkZAMHHR7sh8JDrwcTGxtKiRQsmTpxIz5496dSpE3PnzmX9+vXHxnBavHgxL7/s7YHXgjaOwmrUcCZQzc/Pp06dOkHnAfGXDpxSzz333ENqairz5s1jzJgxftP4Dtle1PiEoQhp6lhXhjsb4DRgtoh8DGwOlkBEBojIGhFZLyLHNe+LyK0i8qOILBWRr0Wkvbu+n4iku9vSReR8nzTz3GMudV+V5M4WIVkH4IOboHZTeGAzjNn321dlu4HWbQnXToMrXvcfNMCeOq/CevXqxdNPP80555xDr169GDduHKeffjoiwooVK2jbtu2xUW179uzJ5MmTAZg0aRK9evUq1jlr1apFixYt+Pe//w04N/Zly5wx4AoPy17Yvn37aNy4MQBvvvlmsc5fHCEHDlX9naruVdUxOLMCvg4MDLS/iETjzE9+IdAeGFoQGHy8q6qnqmpn4EngWXf9TuBSVT0VGA68XSjdMFXt7L7sv7wkPrkP9m6GK16DanUinZuyIeKUPEzFFagkXMIScq9evdi6dSs9evSgQYMGJCQkHAsIn376KQMGDDi27wsvvMAbb7xBp06dePvtt3nuueLPZzdp0iRef/11TjvtNDp06MDHH38MwJAhQ3jqqac4/fTT2bBhw3HpxowZw+DBg+natSv16tUr9vm9KnJY9WM7inTHmVv8gPu+FtBOVRcF2L8HMEZVL3DfPwigqn6nshKRocB1qnphofUC7AIaqmqWiMwD7lPVkMdJ9zqsepXxwxTnyeveDzoDBlY1wao7eoyC04bCSR3LLj9VXHkfVr1fv3689dZbxZoWtiLwMqy6l6qql4GDPu8PuusCaQz84vM+w11XOGMjRWQDTonjTj/HuQL4TlV9W4vecKup/iwBph8UkREikiYiaTt27AiSzSpq90/w33ugWQ/odV+kc1P+LBoH486CcWfDt//ndO01Vdrs2bMrbdDwykvjeMH0sYAzWq6IlHhGHVV9CXhJRK4GHsapmnJOKNIBeALo75NkmKpuEZFE4APgWuAtP8d9FXgVnBJHSfNZqeTlwAe/B4lyuqvaxEjHu3ctLJ8KS9+FmQ/C7D9Dcn+nFNL6AvhnRxvO3VRZXu4YG0XkTv5XyrgdZxbAQLYAvmNUNHHXBTLZ59iISBPgI5zqq2OVe6q6xf15QETeBc7ET+AwQcx73Jn/e/BEZ9DCqirQU+c1kqBGXeh2i/PKXOUEkB+mOHOCVDsBjuzxf0xrWDdVgJfAcSvwPE6pQIEvgBFB9l8CJItIC5yAMQS42ncHEUlW1YKvZxcD69z1dYBPgNGq+o3P/jFAHVXdKSKxwCXA5x6uwfw033mW4fRrnTGoqrJQSwZJ7aD/o9DnL7BxHiydBCs+DGvWqipVJUDtswkjr110vUzklIlz8w91/1wRGQXMBKKBCaq6QkT+BqSp6nRglIj0BXKAPfyvmmoU0Ap4REQecdf1Bw4BM92gEY0TNMaHmqcq7/Bu+HCE0yX1wicinZuKJzoGkvs6r2CBY+qNzjzrTVLgpE4Qm/Db7REcZ6k8S0hIYNeuXdStW9eCRxlSVXbt2kVCQkLRO7u8DDnSGqcqqYGqdhSRTkCqqo4NkqEZwIxC6x7xWb4rQLqxQKDjdg01z8aHKky/Aw7tgKGfQ1yNotOY4vl5oTPJFEBUrNMzqyCQNE4JyzhLlUGTJk3IyMjAOrOUvYSEBJo0aRLy/l6qqsYD9wOvAKjqD24bQ8DAYcqR9Ddg9X+h/9+hUedI56Zyu2cl7N/qtCNlpMGWdFj2HiyxwnEwBU9um/LPS+CorqqLCxUhAzx6a8qVzNXw2Z+g5fnQ/fZI56ZyCNawDlCrIdS61JkpESA/D3asdgLJf/z1Ojem4vASOHaKSEvcecdFZBCwNSy5MqUn56gzpEh8TRg4zpmz25Sc17aIqGho0MF5WeAwFZyXwDES57mItiKyBfgJKHoKLBNZn/8Fti+Hq/8NiQ2K3t9EVuYqpxeXMeWYl7GqNqpqX6A+0BY4Fzg7XBkzpWDtTOcJ6G63Qev+Re9vykbA8ZSiYHwfWPlxmWbHGK+KLHG4Y1KNxBku5GOcLrAjgXuBH4BJ4cygKaYD22DabdDgVOj310jnxvgKVM21/1d4/1qYcp0z1e15D9l8IaZcKnKQQ3f49D04s/31AZIAAe5S1aXhzmBpqDKDHNrzARVfbpYzV8h3b0GrflVr1GJT7gQa5DCUNo5T3OHNEZHXcBrEm6nq0VLOoykpez6g4ouJh0ufh4ad4dMHYPx5MORda/cw5UoobRw5BQuqmgdkWNAwJoxE4Iyb4Pr/QtZBt91jeqRzZcwxoQSO00Rkv/s6AHQqWBaR/eHOoDFVVrPucMuXTmljyrXwxaPO8yDGRFiRgUNVo1W1lvtKVNUYn+VaZZFJY6qsWo3ghhnOoJTzn4b3hsCRvZHOlanibCIGY8q7mHhIfcEZKuaTe+GJk4/fxzpAmDJkjxFXFrnZzsRM/pRwHmZTDojAGb8PvN06QJgyZCWOymL5VNB8uOYDaNU30rkxxlRiVuKoDFRhwQuQ1AFa9ol0bkykbPzS+SwYE2YWOCqD9V9A5kroeYdTpWGqprdS4fX+sG62BRATVhY4KoMFz0FiI+h4RaRzYiLpoqedYUsmDYJXe8Oq/0J+fqRzZSohCxwV3a9L4aevoPutEBMX6dyYcAvU0aFGEpx5M9z5vdMD6+heeH8YjDsbln9oz3+YUmWN4xXdghcgLhG6Xh/pnJiyUFSX25g46HIdnHa1M33t/Kdh6g1Qr7UzcOKsP9t4ZqbEwho4RGQA8BwQDbymqo8X2n4rzki7ecBBYISqrnS3PQjc5G67U1VnhnLMKmXvz7DiI+hxOyTUjnRuTHkSHQOnXQWnDnKGaf/qafjolsD7h9Kd1wbRNK6wBQ4RiQZeAvoBGcASEZleEBhc76rqOHf/VOBZYICItAeGAB2ARsDnItLaTVPUMauOhS87jeHdbo10Tkx5FRUNHS+H9gNh7acw+erA+859DOJqQlwNiE/0Wa7plGptEE3jCmeJ40xgvapuBBCRycBlwLGbvKr6jnVVA3daWne/yaqaBfwkIuvd41HUMauMI3sg/U3oOAhqN4l0bkx5FxUFbS8Ovs+XT5RNXkyFF87A0Rj4xed9BtCt8E4iMhK4B4gDzvdJu7BQ2sbucpHHdI87AhgB0KxZM++5L+/SJkDOIacLrjGl4ZE9zmcq6yBkH4LsAz7LB525642hHDSOq+pLwEsicjXwMDC8lI77Ks4c6aSkpFSuTu25WbDoFWh5PpzUMdK5MZVFVJRTRRWf6H+7BQ7jCmd33C1AU5/3Tdx1gUwGBhaR1usxK6cfpsDB7dDzzkjnxFQ0wbrzGhOicJY4lgDJItIC5+Y+BPhNy5yIJKtqQXeMi4GC5enAuyLyLE7jeDKwGGfK2qDHrPTy850uuCedCqf0jnRuTEVTkt5PNZL8N4THVCv+MU2FFLbAoaq5IjIKmInTdXaCqq4Qkb8Baao6HRglIn1xZhncg1tN5e43BafROxcY6c4+iL9jhusayqX1s2HnGrh8vA0vYsqWv6Dz2YOw8P9g1X+g3aVlnycTEaJVYEyblJQUTUtLi3Q2SscbF8OeTXDXUoiOjXRuTFWXmw0T+sPujXDr11CnEnZEqcJEJF1VUwqvtyFHKpIt6bD5a+h+mwUNUz7ExMGgCU4V6tSbIC8n0jkyZcACR0Wy4AWIrw1dS6XjmTGl48RTIPU5yFgMc/8e6dyYMmCBo6LY/ZMzdETK9YG7SxoTKR2vgC7D4et/OsP8m0ot4s9xmBAtfBkkGrrdFumcGOPfgMfhl8XOmFi3fgOJDUrnuDZGVrljJY6K4PBu+P5t6HQl1GoY6dwY419cdRg80Xna/MPfl95Q7jZGVrljgaMiWPI65ByGHqMinRNjgktqCxc96cwRM//ZSOfGhIkFjvIu5ygsfgVa9YMG7SOdG2OKdvq1cOpgmPcYbF4Q6dyYMLDAUd79MBkO7YCzbHgRU0GIwCX/hBOawwe/d6pai2P7Cpg0uFSzZkqHBY7yLD8fFrwIDU+D5r0inRtjQhefCIPecL70TLsNvDxovG8LTBsJL58FPy8Kvm9uVsnyaYrFAkd5tvYz2LXOGczQhhcxFU2jztDvUedzvPDlovc/shc+HwMvdIEfp0CPkc4ICcEGYJwy3Hl63ZQp645bni14Hmo3c2ZvM6Yi6naL01A++xFo1h0adzl+n9wspwPIV086E5R1ugrOewhOONnZHqjL7eLxMOM+Z071wRNtNIUyZGNVlSfWX91URod3w1MtQfOP3xZfC6rVgb0/wynnQb+/OlWzoVr4Mnw2Gjpc7gz8GW3fhUtToLGq7Ldcnlh/dVMZVT/Rf9AAyNoPdU6Gaz6EVn28H7v7bc74WLP/7JQ4Br7szLNuwsoChzEmsm75ypl9sLjOuhPysmHOoxAVC6kvlOx4pkgWOIwxkVUaN/lz7nNKHl8+7lRXXfKv0ulQYtXHflngMMZUDr1HOyWPr591Sh4XPVXy4GHVx35Z4CgP8vPhizGRzoUxFZsI9HnECR7fvui0eVzwmHVlDwMLHJGWc9R5QGrFh87czblHjt8nWD92YyqCQPOVl/ZnWwT6j4X8XGdK2+hY6PtXb8Ej66DThXjdrNLNWyUS1sAhIgOA53DmB39NVR8vtP0e4Pc484rvAG5U1c0ich7wT59d2wJDVHWaiEwEzgX2uduuV9Wl4byOsDm8GyYPg58XOB/us+6yb0emcirL9gARZ4j3vBz45jmIjofzHwq8vyrs2uAEinWzYPM3TqklrmbZ5bmCCVvgEJFo4CWgH5ABLBGR6aq60me374EUVT0sIrcBTwJXqepcoLN7nBOB9YBv+L9fVaeGK+9lYs8meGcQ7N0MV7wOpw6KdI6MqTxE4KKnnQDw1ZPOq7CEOs7DhutmwZ6fnHX12sCZIyC5PzTrAWPrl2m2K4pwljjOBNar6kYAEZkMXAYcCxxugCiwELjGz3EGAZ+q6uEw5rVsbfkO3r3S+UZ07TRoflakc2RM5RMVBZc+78xl48/RvfDdW9DiHGd4k+R+zsCMvgJVscXVKO3cVijhDByNgV983mcA3YLsfxPwqZ/1Q4DCA/v/XUQeAb4ARqvqcSOdicgIYARAs2bNPGQ7zNZ8ClNvhBr14PoZUL91pHNkTOVVVFffB36C2GqBtxeuYlN1RvxdPhVW/QfaXVryPFZA5eIpGRG5BkgBniq0viFwKjDTZ/WDOG0eZwAnAg/4O6aqvqqqKaqaUr9+OSluLh4Pk6+G+m3g919Y0DAm0oIFDX9E4LIXoXEKfDgCfl0almyVd+EMHFuApj7vm7jrfkNE+gIPAal+Sg5XAh+pak7BClXdqo4s4A2cKrHyLT/fGeRtxn1O3en1n0BN6yllTIUUWw2GvAvV68J7Q2D/1kjnqMyFs6pqCZAsIi1wAsYQ4GrfHUTkdOAVYICq+nuiZihOCcM3TUNV3SoiAgwEloch78UX6ElTgJSb4MInbSA2Yyq6xAYwdDJMuMAJHjd86sy5XkWErcShqrnAKJxqplXAFFVdISJ/E5FUd7engJrAv0VkqYhML0gvIs1xSixfFjr0JBH5EfgRqAeMDdc1FEuwJ0ovfsaChjFlLdCzIiV9huSkjnDFa7B1GUy71alZqCJsWPXSNqZ2kG37Am8zxlRMC16EWQ9Br/ugz58jnZtSZcOqG2NMOPQYCTvXwPynoV5rOO2qkh+znA+uWC56VRljTIUlAhc9A817wfRR8PPCkh+znA+uaIHDGGNKKiYOrnwLajd1hhHasynSOQorCxylLVwNccaY8q36iXD1FGeAxXeHwNH93o+Rlwsb5ha9X4RZG0dpKwf1j8aYCKnXyil5vHO5M0LE0MlF96TMy3FG4105DVb9F47sLpOsloQFDmOMKU2nnOt0vf/PXfBo3eO310iCu1e4weIjWP0JHNnjjMbbegB0GAjv+xu2r/ywwGGMMaWt6/VO4PDnUCY83QqO7oP4WtDmQmh/GbTsA7EJzj6BBldMqBOuHHtigcMYY8pam4ug/UBoeR7ExB+/vXCVd/YhePks0Dxnoqn4yM4VYo3jxhhT1n43DtoM8B80/ImrAQP/D/b+4ox7F2EWOIwxpiI4uSd0vx3SXo94zysLHMYYU1H0+TPUbQUfjyped99SYoHDGGPCIRzPdMVWg4Hj4MCvzvhYEWKN48YYEw7heqar6RnQ8w745jlol+pMeVvGrMRhjDEVTe8/Qf22MP0O5xmQMmaBwxhjKprYBBj4MhzMhM/+VOant8BhjDEVUeMu0OseWPYurPm0TE9tgcMYYyqqc/4IDTo6T6kfLrsxrixwGGNMRRUT51RZHd4Fn/6xzE5rgcMYYyqyhp2ckseP/4aV08vklGENHCIyQETWiMh6ERntZ/s9IrJSRH4QkS9E5GSfbXkistR9TfdZ30JEFrnHfF9E4sJ5DcYYU+71ugdO6gT/vRsO7Qz76cIWOEQkGngJuBBoDwwVkfaFdvseSFHVTsBU4EmfbUdUtbP7SvVZ/wTwT1VtBewBbgrXNRhjTIUQHeuMf3V0H3xyb9hPF84Sx5nAelXdqKrZwGTgMt8dVHWuqh523y4EmgQ7oIgIcD5OkAF4ExhYmpk2xpgKqUEHOO9BZ0Ko5R+G9VThDByNgV983me46wK5CfDtU5YgImkislBEBrrr6gJ7VTW3qGOKyAg3fdqOHTuKdQHGGFOh9LwLGnVxSh0H/cznUUrKxZAjInINkAKc67P6ZFXdIiKnAHNE5EdgX6jHVNVXgVcBUlJStDTza4wx5VJ0DOzZ5Ew/+3Tyb7fVSCq1YVDCWeLYAjT1ed/EXfcbItIXeAhIVdWsgvWqusX9uRGYB5wO7ALqiEhBwPN7TGOMqbICzVnub0bBYgpn4FgCJLu9oOKAIcBv+oqJyOnAKzhBI9Nn/QkiEu8u1wPOAlaqqgJzgUHursOBj8N4DcYYYwoJW+Bw2yFGATOBVcAUVV0hIn8TkYJeUk8BNYF/F+p22w5IE5FlOIHicVVd6W57ALhHRNbjtHm8Hq5rMMYYc7ywtnGo6gxgRqF1j/gs9w2QbgFwaoBtG3F6bBljjIkAe3LcGGOMJxY4jDGmMgnHzIOFlIvuuMYYY0pJuGYe9GElDmOMMZ5Y4DDGGOOJBQ5jjDGeWOAwxhjjiQUOY4wxnogzikflJiI7gM3FTF4PKO7MKJFKG8lz2zVXjLSRPLddc8VIC85gs/WPW6uq9gryAtIqWtqKmm+7Zvt92TWXn7TBXlZVZYwxxhMLHMYYYzyxwFG0Vytg2kie2665YqSN5LntmitG2oCqROO4McaY0mMlDmOMMZ5Y4DDGGOOJBY4gRGSAiKwRkfUiMtpDugkikikiy4txzqYiMldEVorIChG5y0PaBBFZLCLL3LR/Lcb5o0XkexH5bzHSbhKRH93ZHNM8pq0jIlNFZLWIrBKRHiGma+Oer+C1X0T+4OG8d7u/q+Ui8p6IJHhIe5ebbkUo5/T3uRCRE0Vktoisc3+e4CHtYPfc+SKS4vG8T7m/6x9E5CMRqeMx/aNu2qUiMktEGoWa1mfbvSKi7vTQoZ53jIhs8fl7X+TlvCJyh3vdK0TkSY/X/L7PeTeJyFIPaTuLyMKC/w0R8TsZXYC0p4nIt+7/1n9EpFaAtH7vHaF+xjwJRx/fyvACooENwClAHLAMaB9i2nOALsDyYpy3IdDFXU4E1no4rwA13eVYYBHQ3eP57wHeBf5bjLxvAuoV8/f9JvB7dzkOqFPMv9k2nIeWQtm/MfATUM19PwW4PsS0HYHlQHWc6Qk+B1p5/VwATwKj3eXRwBMe0rYD2gDzgBSP5+0PxLjLTwQ6b5D0tXyW7wTGhZrWXd8UZ1rpzYE+MwHOOwa4L4S/j7+057l/p3j3fZKX9IW2PwM84uHcs4AL3eWLgHke0i4BznWXbwQeDZDW770j1M+Yl5eVOAI7E1ivqhtVNRuYDFwWSkJV/QrYXZyTqupWVf3OXT6AM1974xDTqqoedN/Guq+Qez+ISBPgYuA1T5kuIRGpjfMP8zqAqmar6t5iHKoPsEFVvYwSEANUE5EYnCDwa4jp2gGLVPWwquYCXwKXB0sQ4HNxGU7QxP05MNS0qrpKVdcUldEAaWe5+QZYCDTxmH6/z9saBPicBflf+Cfwx0DpikhbpABpbwMeV9Usd5/M4pxbRAS4EnjPQ1oFCkoKtQnwOQuQtjXwlbs8G7giQNpA946QPmNeWOAIrDHwi8/7DEK8gZcWEWkOnI5Tcgg1TbRbhM4EZqtqyGmBf+H8M+d7SONLgVkiki4iIzykawHsAN5wq8leE5EaxTj/EAL8M/ujqluAp4Gfga3APlWdFWLy5UAvEakrItVxvkU29ZhfgAaqutVd3gY0KMYxSupG4FOviUTk7yLyCzAMeMRDusuALaq6zOs5XaPcarIJHqtdWuP8zRaJyJcickYxz98L2K6qXmZM+gPwlPv7ehp40EPaFfzvS+tgQvicFbp3lPpnzAJHOSUiNYEPgD8U+nYXlKrmqWpnnG+QZ4pIxxDPdwmQqarpxcmv62xV7QJcCIwUkXNCTBeDUzx/WVVPBw7hFKlDJiJxQCrwbw9pTsD5h2wBNAJqiMg1oaRV1VU4VTyzgM+ApUCelzz7OabioYRYGkTkISAXmOQ1rao+pKpN3bSjQjxfdeBPeAg0hbwMtAQ64wT7ZzykjQFOBLoD9wNT3NKDV0Px8AXFdRtwt/v7uhu3dB2iG4HbRSQdpwoqO9jOwe4dpfUZs8AR2BZ+G9mbuOvCTkRicf7wk1T1w+Icw63qmQsMCDHJWUCqiGzCqZY7X0Te8XjOLe7PTOAjnOq+UGQAGT6lo6k4gcSLC4HvVHW7hzR9gZ9UdYeq5gAfAj1DTayqr6tqV1U9B9iDU6fs1XYRaQjg/gxYfVLaROR64BJgmHtDKa5JBKg+8aMlTqBe5n7WmgDfichJoSRW1e3ul6N8YDyhf8bA+Zx96FbpLsYpWfttmA/ErdK8HHjfSzpgOM7nC5wvNyHnW1VXq2p/Ve2KE7A2BMmfv3tHqX/GLHAEtgRIFpEW7rfZIcD0cJ/U/Qb0OrBKVZ/1mLZ+Qe8YEakG9ANWh5JWVR9U1Saq2hznWueoakjfvt3z1RCRxIJlnMbXkHqVqeo24BcRaeOu6gOsDPXcruJ8C/wZ6C4i1d3fex+ceuGQiEiS+7MZzs3kXY/nB+czNdxdHg58XIxjeCYiA3CqJVNV9XAx0if7vL2M0D9nP6pqkqo2dz9rGTgNuttCPG9Dn7e/I8TPmGsaTgM5ItIapxOG15Fj+wKrVTXDY7pfgXPd5fOBkKu5fD5nUcDDwLgA+wW6d5T+Z6ykreuV+YVTb70WJ8I/5CHdezjF6Bycf4ybPKQ9G6co+QNO9cdS4KIQ03YCvnfTLidAr48QjtMbj72qcHqfLXNfK7z8vtz0nYE0N+/TgBM8pK0B7AJqF+Na/4pz01sOvI3b4ybEtPNxAtwyoE9xPhdAXeALnBvJ58CJHtL+zl3OArYDMz2kXY/ThlfwGfPbKypI+g/c39kPwH+AxsX5XyBIT7wA530b+NE973SgoYe0ccA7br6/A873cs3u+onArcX4O58NpLuflUVAVw9p78K5D60FHscd8cNPWr/3jlA/Y15eNuSIMcYYT6yqyhhjjCcWOIwxxnhigcMYY4wnFjiMMcZ4YoHDGGOMJxY4jPFIRPLktyPyjnbXzxNnNOVlIvJNwXMpIhInIv8SZ5TldSLysTsuWMHxThKRySKywR2uZYaItBaR5nL86K5jROQ+d7m7O3zGUnFGFB5Thr8GU4XFRDoDxlRAR9QZ1sWfYaqa5o7V9RTOMCiP4QwV0UZV80TkBuBDEenmpvkIeFNVh4AzjDbOeEK/HH/433gTuFJVl4lINM5IucaEnQUOY8LjK+AP7thMNwAtVDUPQFXfEJEbcZ4gViBHVY89Dazu4H/uQHXBJOE8LIZ7bK9P2xtTLBY4jPGumvx2Ep9/qGrhsYsuxXnCuRXwsx4/UGUa0MFdDjawZMtC5zoJZ3RVcIYmXyMi83AGWnxTVY+GehHGFJcFDmO8C1ZVNUlEjuAMpXEHUNLZ1jb4nsu3HUNV/yYik3DGBbsaZ7yu3iU8nzFFssBhTOkapqrHps0Vkd1AMxFJVGdynQJdgYLpeQcV92SqugF4WUTGAztEpK6q7iru8YwJhfWqMiaMVPUQTiP2s24DNiJyHc5sg3PcV7zvxFci0klEehV1bBG52Gc+iWSc+UD2lu4VGHM8CxzGeFetUHfcx4vY/0HgKLBWRNbhzOL2O3XhjHLb1+2OuwL4B85MbUW5FqeNYynOqLHDChrgjQknGx3XGGOMJ1biMMYY44kFDmOMMZ5Y4DDGGOOJBQ5jjDGeWOAwxhjjiQUOY4wxnljgMMYY48n/A69wHWo+JstmAAAAAElFTkSuQmCC\\n\",\n      \"text/plain\": [\n       \"<Figure size 432x288 with 1 Axes>\"\n      ]\n     },\n     \"metadata\": {\n      \"needs_background\": \"light\"\n     },\n     \"output_type\": \"display_data\"\n    },\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 229 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"value1 = ft_recall_10\\n\",\n    \"value2 = cs_recall_10\\n\",\n    \"name = \\\"Recall@10\\\"\\n\",\n    \"\\n\",\n    \"length = len(value1)\\n\",\n    \"plt.plot(range(length), value1, \\\"-s\\\", label=\\\"Finetune\\\")\\n\",\n    \"plt.plot(range(1, length), value2, \\\"-s\\\", label=\\\"w/o Pretrain\\\")\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"plt.xticks(range(length))\\n\",\n    \"plt.ylabel(name)\\n\",\n    \"plt.xlabel(\\\"EPOCHS\\\")\\n\",\n    \"plt.legend()\\n\",\n    \"plt.show()\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 154,\n   \"id\": \"attached-printer\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"image/png\": \"iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA1E0lEQVR4nO3deXgUVfbw8e/JHiBhC2HXALIICgEiIAoyIMiog7soqCDOuAsuPxVHR3F5x93RGR13XFFQVMAVccENEQKiEkAFRA3DEhANCNnP+0dVoA3dne5OL1nO53n6SXdV3aqTTqdO17237hVVxRhjjKkqLtYBGGOMqZ0sQRhjjPHKEoQxxhivLEEYY4zxyhKEMcYYrxJiHUC4ZGRkaFZWVqzDMMaYOmXZsmXbVLWVt3X1JkFkZWWRm5sb6zCMMaZOEZEffa2zKiZjjDFeWYIwxhjjlSUIY4wxXlmCMMYY45UlCGOMMV7Vm15MxtQ2ObctYNuukv2WZzRJIveGkfXuuKb+sQRhjB81Odl6K+dveW04rjGeLEGYWi+W34iDOdmWllews6iMwj2lFBaV+t3v5+u2k5IYR0piPMkJzk/P5/6O++vuEvaUlrO7pJw9JeUUVT4vdV7Hkl011S+WIEytF+1v4hUVyi+7SyjYWex33yf/9zMKi8rYWVRK4Z4y9pQGfnI+8/HFAW9bVfYtC0Iu+9dncjmkfTqHtm/KIe2b0jo9Zb9t6uLVS129aqrtic0ShKnVqpvQ6v9e/oomyQmkpSTQJDmBxh7PmyQn+D1xPPv5BrYWFlOws5iCXcVs3VlEwc5itu0qobyi+om0GiUl0Do9hfSURNJTE0hPSSQtJYH01ETSUxL567O+7+x/4W8DKS6toLisnKLSCopKnSuB4rIKikor+Nd73/kse+PxPWmUFE9qUjypic7PRknOFUhqYjzD7/3IZ9kftu3i/TVbqHxbW6Ulc0g7J2H0at+UQ9s3rfZku7ukjO27SijYVcy2ncVs/71k709/PvqugA7NU2nfLJWUxPj91gd7slRVft1dypadRX6PW5vFqhoyUJYgTK3z6+4SPlu7nU++L+CT77f53XbR2m3sLC5jV3EZwU6OeOPcPOIEMpok0yrNefRsm+48b5JMZnoKF89Y7rP8838dGNwBPQzukuF3vb8EMenITiEf9/2rhvF7cRmrNhWycuNvrNzo/PzouwICyIn0vPEddvuoxkpL8X86mTB9yd7nGU2SaN+8ER2apTpJo3mq35PlIx+tY0thEVsLi9lSWMSWnUVsKSympKyi2phPePBTurdJo3ubdA5uk0b3Nmm0bJK8d31NT7Shlq/uy8/vxWU0SopHRLyuj8ZVkyUIExX+/okWTR3B8p928Mn3BXz6/Ta+3vgbqs4JZ3CXlmz8dY/P/S66bgTg/LPtLinn9+IyJ2EUOUlj/BNf+Cy79PqjadE4ifg47/+ANZXRJMnn7xxJ1R23cXICh2W14LCsFnvX7SkpZ/XmQvI2/sY/5ub53PeZAw4go0kyLZsk0cr9Wfk6OSGerKlv+iz70gWHs/HX3WzcsYf8HXvY+OseVm8qZMHqLdWe6O94ew1NkhPITE+mdVoK/Q9oTuv0FDLTU2iTnsIlL/hO5I2TE3h/9VZeys33eC+S6dEmjR5t0mp8ovVXftG6bU5CcxNb1QTnT6+b5hMnTvxpyQk0qXKVHA2WIExU+Psnyr7lXXaXlBMfJ/Tt2IwpI7oypGsr+nRoSkJ8nN+TTiURobH7z5MZYEyt0pKr3aYmJ/maXOZH+7ipSfH0O6A5/Q5o7jdB/OP4nkHvu9KATi2AFvstr6hQtv1ezID/977PsitvPoYmyb5PV5e84Pu4L/xtEAAFO4v5dvNO1mwuZM3mnXy7eSfPLfY5Th0Ao/71EXEiiAgCxMXxx9fVfLcY9/i+LyiNk+Jp3TTlDwnu0Y/X+yz792N7sKvoj194dhWXsbOojE2/RadazRKEiajS8gp+3P67321O6deBI7tmcHiXlqSnJO63PlbfxKFmJ/m6eNyaCuVvFRcnZKbt31juyV9yCPS4ldWIR3bdV71XXqF0+ftbPvfbOaMJilKhzlWq58+KAOo0X/jbQFqnp9A6PcXr7+AvQZw/tIvffQfyxammLEGYgPmrJvr02uGsL/idtQW7WLtlJ99v3cX3W3exYdvvlFVTuX3riYf4XR+rb+INVayummoi1ONWV734yNn9q92HvxN1dW1Ntf3zaQnCBMxfNVHPG9/Z28gZJ3Bgy8YclNmEkT1b0zWzCVe+9FUUI92nrn4Tj6VYvWe1/WQZCbX9y48lCBOQ6hoRLx3ela6ZTTgoswmdMhrv140xVgnC1B11NTHFKrFF4/2yBGH82lJYxIwvfuKFL37yu92VI7v5Xd8Qvx2auqGmJ9r6fJVqCcLsR1VZumEHz3y+gfkrN1OuyrBurfjw24KQ91mf/4mMqa8sQZi9dpeUMXfF/3hm0QbWbN5JekoC5x6RxVmDDuTAlo2j0mvCGFN7WIJoYHz1REpNjCMxPo7CojIObpvOHScfygnZ7UlN2teWYNVExjQsEU0QIjIaeACIB55Q1Tu8bHM6MA1Q4CtVHecunwDc4G52m6o+E8lYGwpfPZH2lFYw4uDWTBicRc6Bzb3e3m/VRMY0LBFLECISDzwEjATygaUiMk9VV3ls0xW4DjhCVXeISKa7vAVwE5CDkziWuWV3RCrehmDbLv+39j84rl+UIjHG1AWRvIIYAKxV1fUAIjITOAFY5bHN34CHKk/8qrrVXX4MsEBVf3HLLgBGAy9GMN56Z/uuYpb88Aufr9/O4vXb+W7LrliHZIypQyKZINoDP3u8zgeqDn/ZDUBEPsOphpqmqu/4KNu+6gFE5HzgfIADDjggbIHXZv7uZn73iqNY8sN2Fq//hc/XbefbLTsBaJQUT05WC07s25673vk22iEbY+qoWDdSJwBdgWFAB+BjETk00MKq+hjwGEBOTk6Qgz3XTf7uZu53qzORTGpiPDlZzRmT3Y5BnVvSu0NTEuPjACxBGGMCFskEsRHo6PG6g7vMUz7whaqWAj+IyHc4CWMjTtLwLLswYpHWE/83qhuHd2nJoe2bkZQQ53Ub64lkjAlUJBPEUqCriHTCOeGfAYyrss0c4EzgKRHJwKlyWg+sA/4pIs3d7UbhNGYbPy4d3rXabawnkjEmUBFLEKpaJiKXAvNx2hemq2qeiNwC5KrqPHfdKBFZBZQDV6vqdgARuRUnyQDcUtlg3ZD9urt2z69rjKlfItoGoapvAW9VWXajx3MFrnQfVctOB6ZHMr665Kftu5n49JLqNzTGmDDxXlFtapUVP//KyQ9/xvZdJTRN3X9CHbA2BGNM+MW6F5Opxvy8zUyZ+SWZaSk8de5hdGnVJNYhGWMaCEsQtdj0T3/g1jdX0adDM56YkENGk+rnUDbGmHCxBFELlVcot725iqc+28AxvVpz/9i+fxg0zxhjosESRC2zp6ScKTO/5N1VW5h0RCeuP+7gaufNNcaYSLAEUYts21XMec/k8nX+r9z0l56ce0SnWIdkjGnALEHUEusKdjHxqSUU7Czm0bP6M6pXm1iHZIxp4CxBxICvAfcEeO2SI8ju2CzqMRljTFV2H0QM+BpwT8GSgzGm1rAEYYwxxitLEMYYY7yyBGGMMcYrSxBR9uVPNq22MaZusAQRResKdjHp6aX4uu/NBtwzxtQm1s01SrYUFnHOk0uIE+GDq4aRldE41iEZY4xfliCioLColAnTl7Bjdwkzzx9kycEYUydYFVOEFZWW87dnclm7dRePnNWf3h2axTokY4wJiF1BRFB5hXLFrBV88cMv3D82m6HdWsU6JGOMCZhdQUSIqjJtXh5vr9zMDccdzIl928c6JGOMCYoliAj5zwdreW7xj1wwtDN/HdI51uEYY0zQLEFEwItLfuK+Bd9xct/2XDu6R6zDMcaYkFiCCLN38zZz/WvfcFS3Vtx5am/ibLIfY0wdZQkijHI3/MJlL37Joe2b8t/x/UiMt7fXGFN32RksTL7bspNJTy+lXbNUpk88jMbJ1kHMGFO32VksRL4m/UmIE1o2SY5BRMYYE152BREiX5P+/LK7NMqRGGNMZFiCMMYY45UlCGOMMV5ZgjDGGOOVJYgQbNj2e6xDMMaYiLMEEaSi0nIunrEcX7e/2aQ/xpj6wrq5Bunm1/NYtamQ6RNzGN6jdazDMcaYiLEriCC8siyfF5f8zMXDulhyMMbUe5YgArRmcyHXz/mGgZ1acOXIbrEOxxhjIs4SRAB2FZdx8YzlpKUk8p9xfUmwMZaMMQ2AnemqoapMfeVrNmz7nX+f0ZfMtJRYh2SMMVFhCaIazy3+kTe+3sRVo7pzeJeWsQ7HGGOiJqIJQkRGi8i3IrJWRKZ6WT9RRApEZIX7+KvHunKP5fMiGacvK37+lVvfWMXwHplcdFSXWIRgjDExE7FuriISDzwEjATygaUiMk9VV1XZdJaqXuplF3tUNTtS8VVnx+8lXDJjOZlpKdx3eh+b+McY0+BE8gpiALBWVderagkwEzghgscLm4oK5cqXVrB1ZxH/Hd+PZo3s5jdjTMMTyQTRHvjZ43W+u6yqU0TkaxGZLSIdPZaniEiuiCwWkRO9HUBEzne3yS0oKAhb4A9/tI4Pvy3gH8f3pE/HZmHbrzHG1CWxbqR+HchS1d7AAuAZj3UHqmoOMA64X0T2awRQ1cdUNUdVc1q1ahWWgD5ft5173/2Wv/Rpx9mDDgzLPo0xpi6KZILYCHheEXRwl+2lqttVtdh9+QTQ32PdRvfnemAh0DeCsQKwtbCIy178kqyMxtx+8qGIWLuDMabhimSCWAp0FZFOIpIEnAH8oTeSiLT1eDkGWO0uby4iye7zDOAIoGrjdliVlVdw2Ytf8ntxGY+c1Z8mNqe0MaaBi9hZUFXLRORSYD4QD0xX1TwRuQXIVdV5wGQRGQOUAb8AE93iBwOPikgFThK7w0vvpxrzNa/0uMcXk3vDyHAfzhhj6pSIfk1W1beAt6osu9Hj+XXAdV7KLQIOjWRs4HteaV/LjTGmIYl1I7UxxphayhKEMcYYr6wl1kTH3V3h9637L2+cCVd/H/14jDHVsisIEx3ekoO/5caYmGvQCcLX/NE2r3QYlZXA/76MdRTGmBA06Com68oapOqqicrLoGCNkxD+t9z5uSUPyq1XmDF1UYNOECZI/qqJnhgJm7+Bsj3OsuR0aNsHBl0E7frCyxN977esGBKSwx6uMaZmLEGY8IhLgJxJTjJo1xdadIY4jxpMfwni8RFwyhOQ2SPiYRpjAhdQghCRY4AT2Tca60Zgrqq+E6G4TG1SXgorX/G/zaS3/a9vnOn9CiSlKez8Hzx2FBzz/yDnPLAxsIypFapNECJyP9ANeBZnyG5wBt6bLCJ/VtUpkQvPxFTxLlj+LHz+EBTmV7+9P/66su7cAnMugjevgu/fgxMehMYZNTueMabGAunFdKyqHquqM1X1U/cxEzgOODbC8ZlY2FUAH9wG/+oF86+D5gfCuJcid7y01jB+NhxzO6x7Hx4eDGvfj9zxjDEBCaSKqUhEDlPVpVWWHwYURSAmEyvb18HnD8KKF5yG4x7HwRGXQ8fDnPW+qokaZ9b82HFxcPjF0GkovHIePH8yDLoEjr7JGrCNiZFAEsRE4GERSWNfFVNH4Df2jb5q6gpfXVXjk6Gi1Gls7nMmDL4MMrr+cZto3PHc5hA4fyEsuBEWPwQ/fASnPGkN2MbEQLUJQlWXAwNFpA0ejdSqujmikZnI8NVVtbwYjrwCBl4IaW2iG1NVialw7N1w0NEw52L470Dv29kwHcZEVEB3UrvJAVVdBvwEDBaRnpEMzMTA0dNinxw8dTsGLlrke70N02FMRAXSi+kCYKrzVO7EqVZaCdwuInep6pORDdE0aGmtYx2BCaPS0lLy8/MpKrLmy2hLSUmhQ4cOJCYmBlwmkDaIS4FeQCrwI3CQqm4WkebAh4AlCBM7W/Kgda9YR2EClJ+fT1paGllZWTbnexSpKtu3byc/P59OnToFXC6QKqZSVd2tqtuBdZVtD6q6A9DQwjUx8W09vK/x4cHw7Imw9j1Q+zjWdkVFRbRs2dKSQ5SJCC1btgz6yi2QBKEiUnlNcpzHAVMCLG9qg19/gtcucHopeROOrqqxMOJG2Loanj/FSRZfPu900TW1liWH2AjlfQ+kiukk3CsFVfW8nbYlcFXQRzTRV1YCL58LFeVwyRJo2SXWEQXH3/0XQ66Cwy+DlbNh0YMw9xJ4/xYY8Ddn2I5GLaIfr6nV4uPjOfTQfVPez5kzh3HjxrFokZ8OEX48/fTTjBo1inbt2oUrxFojkG6uP3m+FpGWwA5V3YgzJpOp7d6bBhtz4bRn6l5ygOq7siYkQfY45/6N9Qudm/0+uA0+vtdZXznCrCfrIlvr5dy2gG279h8qPqNJUo2G6k9NTWXFihV/WBZqcgAnQRxyyCH1MkEE2s21uYg8KCIfAQ8Bb4vIdBFpHNnwTI2tfsO54WzA+dDrxFhHE1ki0OVPcNYrcPFiOPQU78kBrItsHeAtOfhbXhNNmjQBYOHChQwbNoxTTz2VHj16MH78eNRt21q2bBlHHXUU/fv355hjjmHTpk3Mnj2b3Nxcxo8fT3Z2Nnv27CErK4tt27YBkJuby7BhwwCYNm0akyZNYtiwYXTu3Jl///vfe4///PPPM2DAALKzs7ngggsoLy8P++8YikC6uTYD3gL+rqqXeiz/E3CHiLwE5KnqLxGL0oTmlx+cG83a9YVRt8U6mujKPBhOeMhpkzC10s2v57Hqf4UhlR376Odel/dsl85Nf/Hfq23Pnj1kZ2cD0KlTJ1577bU/rP/yyy/Jy8ujXbt2HHHEEXz22WcMHDiQyy67jLlz59KqVStmzZrF9ddfz/Tp03nwwQe55557yMnJqTbuNWvW8OGHH7Jz5066d+/ORRddxNq1a5k1axafffYZiYmJXHzxxcyYMYNzzjknsDcjggJpg/gHcI+qfigizwGDgG1ABvANIMANwJURi9IEr6x43xwMpz1t4xkZ4/JWxeRpwIABdOjQAYDs7Gw2bNhAs2bNWLlyJSNHOlVb5eXltG3bNuhjH3fccSQnJ5OcnExmZiZbtmzh/fffZ9myZRx2mDPm2Z49e8jMrB2dRgJJEENVtbIxuhg4U1VzRaQfcBHwKfBApAI0IXr3Bti0AsbOgOZZsY7GmP1U900/a+qbPtfNuuDwcIezV3Lyvi9T8fHxlJWVoar06tWLzz/3fuXiKSEhgYqKCoD9upX62veECRO4/fbbw/QbhE8gbRApsq9/VD/gK/f5SqCfqlZEJDITurzXYMljzmioBx8f62hqr/LSWEdg6oju3btTUFCwN0GUlpaSl5cHQFpaGjt37ty7bVZWFsuWLQPglVeqmWgLGDFiBLNnz2brVqdd7JdffuHHH38M968QkkASxBJghPv8v8C7IvJPYD7wqIgcBuRFKD4TrO3rYO5l0D7HGVupofN3f8f8v0cvDhO0jCZJQS2PpKSkJGbPns21115Lnz59yM7O3tvzaeLEiVx44YV7G6lvuukmpkyZQk5ODvHx8dXuu2fPntx2222MGjWK3r17M3LkSDZt2hTpXykgotXcfSoinYGXgONUdYuIZACdgfU4CWYeMEFVv410sP7k5ORobm5uLEOIvdIiePJo+PVnuPATaHZArCOqveZf73SHHfMg9Ds71tE0GKtXr+bggw+OdRgNlrf3X0SWqarXFvZA7oNYLyKXAPNE5F1gMVCOM5vcscBVsU4OxjX/Otj8DZw5y5JDdY6+2RnH6c0roVWPfZMiGWP2Cug+CFX9Ajgc+Bg4GDgEJ1EMVtVPIheeCdg3syF3OgyeDN1Hxzqa2i8+AU6dDuntYNZZUFg7LumNqU0C6cUEgNsYvcB9mNpk2/fw+hToONAZm8gEplELOONFeOJoJ0lMfBMSU8K3f1+z99ld3KaOqPYKQkR2ikihx89Cz9fRCNL4UbrHud8hPglOfQriAx/r3QCte8JJjzhDkbx5VXhHhPV1t7bdxW3qiEDaINKiEYgJkK9vpSnNoGn7/Zeb6vUcA0OvgY/vgrZ9YOD5sY7ImFohkKE2/A6HaUNsRJmvb59Fv0Y1jHpnmNvA/85UZ5iOTkNqtr/ysvDEZUwMBdJIvQzIdX9WfTTwfqWm3oiLg5Mfc0a7fXmCM39GKFRh1Tz476Dwxmei6o477mDGjBkBbfv000/TqlUrsrOz6dmzJ48//nhQx5ozZw6rVq0KOsZ58+Zxxx13BF0uGIFUMQU+P50xdVlKutNo/fhwmDkOJr0LSY0CL7/hU1hwk9OekdE9cnE2FDFs5J8/fz4vvfRSwNuPHTuWBx98kK1bt9KrVy/GjBlD69b75lMvKysjIcH76XbOnDkcf/zx9OzZc791/sqNGTOGMWPGBBxjKIKaEc4d9nuAiAytfFSz/WgR+VZE1orIVC/rJ4pIgYiscB9/9Vg3QUS+dx8TgonTmJBlHASnPAGbVzqTDwXSaL15Jcw4DZ4+Dgr/B2P+Axct8n0Xd3ySM3mT8S8Cjfx333333mG2r7jiCoYPHw7ABx98wPjx4wEoLCykpKSEVq1asWHDBoYPH07v3r0ZMWIEP/3k/8oyMzOTLl268OOPP+69w3rgwIFcc801rFu3jtGjR9O/f3+GDBnCmjVrWLRoEfPmzePqq68mOzubdevWMWzYMC6//HJycnJ44IEHeP311xk4cCB9+/bl6KOPZsuWLYBz5XLppc4A2xMnTmTy5MkMHjyYzp07M3v27JDfI08Bd3N1T95TgA7ACpxRXT8HhvvYPh5n7oiRQD6wVETmqWrVa6lZnsOIu2VbADcBOTiz2S1zy+4INF5jQtZtlNNd+P2boW1vOPIK79vt+BE+/Cd8Pcu5+hh5izPvRmKqs97bt9wvHoO3r3bu5P5zZKsHar23pzrtPqF46jjvy9sc6vd9HTJkCPfeey+TJ08mNzeX4uJiSktL+eSTTxg61Pm++9577zFihDO60GWXXcaECROYMGEC06dPZ/LkycyZM8fn/tevX8/69es56KCDAMjPz2fRokXEx8czYsQIHnnkEbp27coXX3zBxRdfzAcffMCYMWM4/vjjOfXUU/fup6SkhMqRIXbs2MHixYsREZ544gnuuusu7r333v2OvWnTJj799FPWrFnDmDFj/rC/UAWcIHCSw2HAYlX9k4j0AP7pZ/sBwFpVXQ8gIjOBE4BAKtuOARZUNoCLyAJgNPBiEPHWPxUVEJcIFV4Gmaurc0rXVkde4Zy83pvmPKpKTHWuAiQOjpgCR14Oqc2r3+/A82HHD7D4v84ou4MuDG/cxq/+/fuzbNkyCgsLSU5Opl+/fuTm5vLJJ5/svbJ45513OPfccwH4/PPPefXVVwE4++yzueaaa7zud9asWXz66ackJyfz6KOP0qKF07fntNNOIz4+nl27drFo0SJOO+20vWWKi33PnT527Ni9z/Pz8xk7diybNm2ipKSETp281/qfeOKJxMXF0bNnz71XGTUVTIIoUtUiEUFEklV1jYj4q2htD/zs8TofGOhlu1PcqqrvgCtU9WcfZa0P59InnORw4sPOFJsmckTghAch71Xv60v3QL9z4KipwXcvHnWb0wj+zlRnSJQex9Y83rqouiuoaU19rzvX91Dg/iQmJtKpUyeefvppBg8eTO/evfnwww9Zu3bt3jGKlixZwsMPPxzUfivbIKpq3NiZdLOiooJmzZr5nYfCWzlwrmKuvPJKxowZw8KFC5k2bZrXMp5DiVc3xl6ggmmDyHdnl5sDLBCRuUBNx6R9HchS1d44d2g/E0xhETlfRHJFJLegoKCGodRyOzY432QPOtqZe9lEXlI1M+qO+U9o957ExcPJjzsz/b1yHmxcHlp8JiRDhgzhnnvuYejQoQwZMoRHHnmEvn37IiLk5eXRo0ePvaOwDh48mJkzZwIwY8YMhgwJrftzeno6nTp14uWXXwacE/hXXzkzJ1QdLryq3377jfbtnc/ZM88EdYqssYAThKqepKq/quo0nFnmngRO9FNkI9DR43UHd5nnPrerauV11hNA/0DLuuUfU9UcVc1p1apVoL9K3aMK8yY71RnH3+98uzV1W1IjGDcLGmfAC2ND71Zbn/mqNq1hdeqQIUPYtGkThx9+OK1btyYlJWXvif/tt99m9Oh9Y5n95z//4amnnqJ3794899xzPPBA6HOjzZgxgyeffJI+ffrQq1cv5s6dC8AZZ5zB3XffTd++fVm3bt1+5aZNm8Zpp51G//79ycjICPn4oah2uO+9G4oMwpl7eqf7Oh042B3Iz9v2CTjVRiNwTu5LgXGqmuexTVtV3eQ+Pwm4VlUHuY3Uy3AmKAJYDvT3d1NevR7ue9nTzlhLx/8LcibFOpqGxV81x7Tfar7/rWvgyVGQ3hYmzYfUZjXfZy1W24f7HjlyJM8++2xI04nWBcEO9x1MFdPDwC6P17vcZV6pahlwKc7EQquBl1Q1T0RuEZHKzruTRSRPRL4CJgMT3bK/ALfiJJWlwC0N9o7t3/Jh/g2QNQT6TYx1NCbcMnvA2OeciZ5eOhvKSmIdUYO2YMGCepscQhFMI7Wox+WGqla4Vwk+qepbwFtVlt3o8fw64DofZacD04OIr/5RhTeuAC136rvjgrptxYRD40zfN2uFS+ejYMy/Yc5F8MblcMJDVo1oaoVgEsR6EZnMvquGi3FmlTOR8vUs+P5dGH0ntLAb2mMiWsNyZ49z7qv46A5o3gmOujo6xzXGj2C+kl4IDMZpT6jssmrDXkbKzi3w9rXQcZBz85Wp/4ZNhd5nwIe3wdeBD/NQ14SrC6YJTijvezATBm0Fzgj6CCZ4qs5UmKV7nL74VrXUMIg4VYmFG51hPtLbQdaRsY4qrFJSUti+fTstW7ZErBotalSV7du3k5IS3IRYwQy10Q2neqm1qh4iIr2BMap6W3ChmmrlvQZr3nDmTc7oGutoTDQlJDmN1nd1ccZ2qqqOz0bXoUMH8vPzqff3LdVCKSkpdOjQIagywbRBPA5cDTwKoKpfi8gLgCWIcPp9G7x1NbTrB4dfWv32pv5Jbe50TPCmjs9GV3kns6kbgqm7aKSqS6oss1lRwu3ta6HoN6cnS3ww+dsYY8IrmASxTUS64IyuioicCmyKSFQN1Zo3YeVsOOoaZ65kY4yJoWC+ol4CPAb0EJGNwA/A+IhE1RDt2QFvXAmtD/U9vLQxxkRRML2Y1gNHi0hjnCuP3Ti9mmo6YJ8BZ36A3wuc8XniE2MdjanNSnYHN9OdMSGqtopJRNJF5DoReVBERuIkhgnAWuD0SAfYIHz/HqyY4cwp0C471tGY2sDfndozz3S6QBsTYdUO1ucO670DZ/a4EUAmIMAUVV0R6QADVacG6/M5124ruHpt9OMxdceKF2DOxXDQCBg7AxKD69duTFX+BusLpIqps6oe6u7oCZyG6QNUtSiMMTYsPufatb7hphrZ46CiDOZdBi+d49wzkZBcfTljQhBIL6a981uqajmQb8nBmBjqdw4cdx98Px9ePhfKvUxBa0wYBHIF0UdECt3nAqS6rwVQVU2PWHTGGO8OO8+ZE/vtq2H2JDh1unVuMGFXbYJQ1fhoBGKMCdLA8505yuf/HV4935nG1G6uNGFkn6ZoKyqsfhtjAnX4JU4V03s3QVwCnPSIM+e1MWFgCSKayophlp97C8M5CY1pOI683LmS+OA2p5ppjI0AbMLDEkS0VFTAaxfADx/DSY9Bn7GxjsjUJ0OvhvIyZ8KhuHg4/gFLEqbGLEFEgyq8c60zjPfIWy05mMgYNtXpAvvJPbD82f3X1/Ghwk302VeMaPjkXljymDN89xGTYx2Nqa9EYPgNvtfX8aHCTfRZgoi05c/BB7fCoac7Vw/GRJLN0mbCyKqYIunbd+D1KdBluDO/g9UJm1h7dChkdHMfXZ2fLbr8ccgOn0PBWBVVQ2MJIlJ+XgIvT4Q2h8LpzzpTSRoTa40y4Kcv4JuXPRYKND9wX+LwORSMVVE1NJYgIqHgW3jhdEhvC+NnQ3JarCMyxnH2q87Pkt2wfS1s+w62fb/v5w8fxzY+U6tYggi33zbCcydDXCKc9So0aRXriExD0zjTdxVRpaRG0La38/BUUQG3NI9sfKbOsAQRTnt2wPOnOHNKn/smtLDJ2U0M1KSdwNrJjAf7NIRL6R548Uznsv2MGdC2T6wjMib8dv8S6whMFNkVRKh89fRITofOR0U/HmPCxVcVFcD0Y5x2teYHRjcmExOWIELl6x+o2AbjM3WcryqqDZ85050+ORLGv2xXyQ2AVTEZYwKTdQRMehfik+CpY2Hte7GOyESYJQhjTOAye8B5C6B5J5hxOnz5fKwjMhFkCcIYE5z0tnDuW9BpKMy9BBbe6QxIaeodSxDGmOClpDvtEH3OhIX/hNcnO8ONm3rFGqlDFcjNSMbUZ/GJcOLD0LQDfHw37NwMpz4FyU1iHZkJE0sQobJBy4zZN8R4ent480q480BnToqqbKC/OsmqmIwxNZdzLpzxovfkADbQXx1lCcIYEx7dR8c6AhNmEU0QIjJaRL4VkbUiMtXPdqeIiIpIjvs6S0T2iMgK9/FIJOM0xhizv4i1QYhIPPAQMBLIB5aKyDxVXVVluzRgCvBFlV2sU9XsSMVnjImyR4dCvwlw6GlOLyhT60XyCmIAsFZV16tqCTATOMHLdrcCdwJFEYzFGBNrFeVOQ/a93WHOJc6kWnb/RK0WyV5M7YGfPV7nAwM9NxCRfkBHVX1TRK6uUr6TiHwJFAI3qOonVQ8gIucD5wMccMAB4YzdGBMKf92/L/wUNi6H5U/DN6/Aiuchsyf0Owd6j4WHBtpUp7VMzLq5ikgccB8w0cvqTcABqrpdRPoDc0Skl6r+YSQ8VX0MeAwgJyfHvooYE2vVncg79Hcex/wTVr4Cy56Bd6bCgpugvNh7GesBFTORrGLaCHT0eN3BXVYpDTgEWCgiG4BBwDwRyVHVYlXdDqCqy4B1QLcIxmqMiabkNOg/Ec7/0Lmy6HdOrCMyXkQyQSwFuopIJxFJAs4A5lWuVNXfVDVDVbNUNQtYDIxR1VwRaeU2ciMinYGuwPoIxmqMiZU2h8Jx98Q6CuNFxBKEqpYBlwLzgdXAS6qaJyK3iMiYaooPBb4WkRXAbOBCVbWprIxpqGwmu5gQrSe9CHJycjQ3NzfWYRhjQjWtqe91qS3g6GnQ92ybNzvMRGSZquZ4W2fvtDGmdvA10GVqC2jV3Rkxdvoo2PRVdONqwGywPmNM7eCvB5QqfDUTFvwDHhsGh/0V/nQ9pDaLVnQNkl1BGGNqPxHIPhMuzYWc82DpE/BgDqx40W62iyBrgzDG1D3/W+Hclb1xGRwwGArWwB4vDdl2k121rA3CGFO/tMuG896DvzwABau9Jwewm+xqyBKEMaZuiotzbra7dFmsI6m3LEEYY+q2xi39r//8v05VVHlpdOKpR6wXkzGmfpt/nfMzsRG07w8HDIKOg6DjYZDSFO7uaoME+mAJwhhTv125Gn5aDD9/4fz85D7QckCc0WR9tVNY+4UlCGNMPeBvmPH0dnDIyc4DoHgXbMyFn76AnxfD1rzoxlqHWIIwxtR9wVQFJTeBzsOcB/gf4qOBs0ZqY4wxXlmCMMYYXzYuj3UEMWUJwhjTsPkaJFDi4NkTIb/hjtBgbRDGmIbNV/vFrz/DM8fDcyfBWa9AxwHRjasWsCsIY4zxpllHmPgWNM5wksSPn8c6oqizBGGMMb40be8kibQ28PwpsOGzWEcUVVbFZIwx/qS3hYlvwjNjYMapMG4WdBoann3X8ru47QrCGGOqk9YGJr4BzQ6EGafD+oXh2W8tv4vbEoQxxgSiSSZMeB1adIYXxsLa92MdUcRZgjDGmEA1aeUkiZZd4cUz4fsFwZVXhS158OHt8NCgyMQYRtYGYYwxwWjcEibMg+dOhJnj4PTnoPto39urwqavYNVcWD0Ptq8FBA48IloRh8wShDHGBKtRCzhnrtP99cWx3rdJbQ59z3ISw68/gcRDpyEw6GLocTykta7140BZgjDGmFCkNoez58CdB3pfv2cHLH7EGRRw6DXQ/dj9JzfyNQptfLJz5SES7qiDYgnCGGNCldrM//qr1/rfxltX1k/vh/dugs/uhyOvCD22MLBGamOMiZTqEog3R0yBXifDezfD2vfCHlIwLEEYY0xtIgInPOjMdjf7PPhlfcxCsQRhjDG1TVJjOGOG83zmWVDye0zCsARhjDE14Wu4cF/LA9WiE5w6HQpWw9xLnEbrKLNGamOMqYlIjpl00AgYcZPTaN02G468PHLH8sKuIIwxpjY7Ygr0Ognevznqw3tYgjDGmNpMBE54CFodDLMnRbXR2hKEMcbUdjFqtLYEYYwxdUEMGq0tQRhjTF1R2Wid9xp89kDED2cJwhhj6pIoNlpbN1djjKlLKhutV82D50/ef30YpyuN6BWEiIwWkW9FZK2ITPWz3SkioiKS47HsOrfctyJyTCTjNMaYOiWpMWi593VhnK40YlcQIhIPPASMBPKBpSIyT1VXVdkuDZgCfOGxrCdwBtALaAe8JyLdVH29I8YYY8ItklcQA4C1qrpeVUuAmcAJXra7FbgTKPJYdgIwU1WLVfUHYK27P2OMMVESyQTRHvjZ43W+u2wvEekHdFTVN4Mt65Y/X0RyRSS3oKAgPFEbY4wBYtiLSUTigPuAq0Ldh6o+pqo5qprTqlWr8AVnjDEmor2YNgIdPV53cJdVSgMOARaKM61eG2CeiIwJoKwxxjRsvqYrrekosh4imSCWAl1FpBPOyf0MYFzlSlX9DciofC0iC4H/U9VcEdkDvCAi9+E0UncFlkQwVmOMqVsiOYqsK2IJQlXLRORSYD4QD0xX1TwRuQXIVdV5fsrmichLwCqgDLjEejAZY0x0icZgEopIyMnJ0dzc3FiHYYwxdYqILFPVHG/rbKgNY4wxXlmCMMYY45UlCGOMMV7VmzYIESkAfqzBLjKAbQ2obCyPXRfLxvLY9jvXjbKxPHZNyh6oqt5vJFNVezhJMrchla2rcdv7Zb9zbS1bl+P29bAqJmOMMV5ZgjDGGOOVJYh9HmtgZWN57LpYNpbHtt+5bpSN5bFrGrdX9aaR2hhjTHjZFYQxxhivLEEYY4zxqsEniEDnzfZRdrqIbBWRlSEct6OIfCgiq0QkT0SmBFE2RUSWiMhXbtmbQzh+vIh8KSJvBFlug4h8IyIrRCTowa9EpJmIzBaRNSKyWkQOD7Bcd/eYlY9CEbk8iONe4b5XK0XkRRFJCaLsFLdcXiDH9Pa5EJEWIrJARL53fzYPouxp7rErPOdtD7Ds3e57/bWIvCYizYIoe6tbboWIvCsi7YI5tse6q9w55zMCLSsi00Rko8ff+9hgjisil7m/d56I3BXEcWd5HHODiKwIomy2iCyu/N8QEZ+zYPoo30dEPnf/v14XkXQv5byeNwL9fAUtEn1n68oDZ5TZdUBnIAn4CugZRPmhQD9gZQjHbgv0c5+nAd8FemxAgCbu80Sc+bwHBXn8K4EXgDeCLLcByKjBe/4M8Ff3eRLQLMS/22acG3wC2b498AOQ6r5+CZgYYNlDgJVAI5zRj98DDgr2cwHcBUx1n08F7gyi7MFAd2AhkBPkcUcBCe7zO4M8brrH88nAI8Ec213eEWdE5x99fW58HHsazvD/Qf8PAn9y/07J7uvMYGL2WH8vcGMQx30X+LP7/FhgYZBxLwWOcp9PAm71Us7reSPQz1ewj4Z+BRHovNleqerHwC+hHFhVN6nqcvf5TmA1XqZV9VFWVXWX+zLRfQTc20BEOgDHAU8EFXQNiUhTnH+MJwFUtURVfw1hVyOAdaoazJ3zCUCqiCTgnOz/F2C5g4EvVHW3qpYBHwEn+yvg43NxAk5yxP15YqBlVXW1qn5bXaA+yr7rxg2wGGfyrUDLFnq8bIyfz5if/4V/AdeEWLZaPspeBNyhqsXuNl5m1fF/XBER4HTgxSDKKlD5rb8pfj5jPsp3Az52ny8ATvFSztd5I6DPV7AaeoIIaO7rSBORLKAvzpVAoGXi3cvfrcACVQ24LHA/zj9tRRBlKinwrogsE5HzgyzbCSgAnnKrt54QkcYhxHAGPv5xvVHVjcA9wE/AJuA3VX03wOIrgSEi0lJEGuF8M+xYTRlvWqvqJvf5ZqB1CPuoqUnA28EUEJH/JyI/A+OBG4MsewKwUVW/Cqach0vdKq7pQVaZdMP5m30hIh+JyGEhHHsIsEVVg5mV53Lgbvf9uge4Lshj5rHvC+ppVPM5q3LeiMjnq6EniJgTkSbAK8DlVb6x+aWq5aqajfONcICIHBLg8Y4HtqrqslDiBY5U1X7An4FLRGRoEGUTcC6rH1bVvsDvOJfDARORJGAM8HIQZZrj/ON1wpmhsLGInBVIWVVdjVM18y7wDrACqNHkVerUA0S1f7mIXI8z+daMYMqp6vWq2tEtd2kQx2sE/J0gk4qHh4EuQDZOUr83iLIJQAtgEHA18JJ7RRCMMwniS4jrIuAK9/26AvdKOQiTgItFZBlO9VGJrw39nTfC+flq6AkipnNfi0gizh95hqq+Gso+3CqaD4HRARY5AhgjIhtwqtSGi8jzQRxvo/tzK/AaTjVdoPKBfI+rndk4CSMYfwaWq+qWIMocDfygqgWqWgq8CgwOtLCqPqmq/VV1KLADp943WFtEpC2A+9NrtUckiMhE4HhgvHvyCMUMvFR5+NEFJyF/5X7WOgDLRaRNIIVVdYv7JagCeJzgP2evulWxS3CulL02kHvjVkOeDMwK4pgAE3A+W+B8gQkmZlR1jaqOUtX+OMlpnY/4vJ03IvL5augJYu+82e430zMAn1OhhpP7jeZJYLWq3hdk2VaVvVFEJBUYCawJpKyqXqeqHVQ1C+f3/UBVA/o2LSKNRSSt8jlOA2jAPbhUdTPws4h0dxeNwJlWNhihfLP7CRgkIo3c930ETt1tQEQk0/15AM6J44Ugjw/O52qC+3wCMDeEfQRNREbjVCeOUdXdQZbt6vHyBAL8jAGo6jeqmqmqWe5nLR+ncXVzgMdu6/HyJIL4nAFzcBqqEZFuOJ0hghnp9GhgjarmB1EGnDaHo9znw4GgJo32+JzFATcAj3jZxtd5IzKfr3C0dNflB06d8nc42fr6IMu+iHP5W4rzD3BeEGWPxLkM/Bqn2mIFcGyAZXsDX7plV+Kjp0UA+xlGEL2YcHp7feU+8oJ9v9x9ZAO5buxzgOZBlG0MbAeahnDcm3FOcCuB53B7uARY9hOcRPYVMCKUzwXQEngf56TxHtAiiLInuc+LgS3A/CDKrsVpZ6v8jHntieSj7Cvu+/U18DrQPtT/Bfz0fvNx7OeAb9xjzwPaBlE2CXjejX05MDyYmIGngQtD+BsfCSxzPydfAP2DLD8F51z0HXAH7kgXVcp5PW8E+vkK9mFDbRhjjPGqoVcxGWOM8cEShDHGGK8sQRhjjPHKEoQxxhivLEEYY4zxyhKEMT6ISLn8cQTZqe7yheKMAPyViHxWeV+HiCSJyP3ijAz8vYjMdce9qtxfGxGZKSLr3KFK3hKRbiKSJfuPRjpNRP7PfT7IHTZihTgj4E6L4ttgGrCEWAdgTC22R53hTLwZr6q57nhUd+MM//FPnCESuqtquYicC7wqIgPdMq8Bz6jqGeAM74wzZs7P++/+D54BTlfVr0QkHmdkV2MizhKEMTXzMXC5O/bQuUAnVS0HUNWnRGQSzl21CpSq6t67Y9UdxM4ddM2fTJybqnD3Hezd58aExBKEMb6lyh8njLldVauOz/MXnDt+DwJ+0v0HXMwFernP/Q2Q2KXKsdrgjAgKzpDZ34rIQpwBA59R1aJAfwljQmUJwhjf/FUxzRCRPThDSFwG1HQGr3Wex/JsZ1DVW0RkBs7YV+NwxqMaVsPjGVMtSxDGhGa8qu6dclVEfgEOEJE0dSZyqdQfqJzW9dRQD6aq64CHReRxoEBEWqrq9lD3Z0wgrBeTMWGgqr/jNCbf5zYkIyLn4Mxe94H7SPacZElEeovIkOr2LSLHecxn0BVnPopfw/sbGLM/SxDG+JZapZvrHdVsfx1QBHwnIt/jzAp2krpwRmU92u3mmgfcjjP7V3XOxmmDWIEzyun4yoZwYyLJRnM1xhjjlV1BGGOM8coShDHGGK8sQRhjjPHKEoQxxhivLEEYY4zxyhKEMcYYryxBGGOM8er/AxJQFOw0o8vUAAAAAElFTkSuQmCC\\n\",\n      \"text/plain\": [\n       \"<Figure size 432x288 with 1 Axes>\"\n      ]\n     },\n     \"metadata\": {\n      \"needs_background\": \"light\"\n     },\n     \"output_type\": \"display_data\"\n    },\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 208 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"value1 = ft_recall_30\\n\",\n    \"value2 = cs_recall_30\\n\",\n    \"name = \\\"Recall@30\\\"\\n\",\n    \"\\n\",\n    \"length = len(value1)\\n\",\n    \"plt.plot(range(length), value1, \\\"-s\\\", label=\\\"Finetune\\\")\\n\",\n    \"plt.plot(range(1, length), value2, \\\"-s\\\", label=\\\"w/o Pretrain\\\")\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"plt.xticks(range(length))\\n\",\n    \"plt.ylabel(name)\\n\",\n    \"plt.xlabel(\\\"EPOCHS\\\")\\n\",\n    \"plt.legend(loc='center right')\\n\",\n    \"plt.show()\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"united-recall\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  }\n ],\n \"metadata\": {\n  \"kernelspec\": {\n   \"display_name\": \"Python 3\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.7.9\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n"
  },
  {
    "path": "finetune/suicide/fine-tune.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"id\": \"wicked-finder\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 202 µs\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"%load_ext autotime\\n\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"id\": \"random-fluid\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 1.05 s\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"import numpy as np\\n\",\n    \"import _pickle as pickle\\n\",\n    \"import pandas as pd\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"id\": \"diverse-vegetation\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"==========LOADING DATA==========\\n\",\n      \"time: 41.8 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(\\\"==========LOADING DATA==========\\\")\\n\",\n    \"age_seq = pickle.load(open(\\\"../data/new_age_seq\\\",\\\"rb\\\"))\\n\",\n    \"sex_seq = pickle.load(open(\\\"../data/new_sex_seq\\\",\\\"rb\\\"))\\n\",\n    \"\\n\",\n    \"util_seq = pickle.load(open(\\\"../data/new_util_seq\\\",\\\"rb\\\"))\\n\",\n    \"code_seq = pickle.load(open(\\\"../data/new_code_seq\\\",\\\"rb\\\"))\\n\",\n    \"date_seq = pickle.load(open(\\\"../data/new_date_seq\\\",\\\"rb\\\"))\\n\",\n    \"label_seq = pickle.load(open(\\\"../data/new_label_seq\\\",\\\"rb\\\"))\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"id\": \"disabled-favor\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 5.04 s\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"import numpy as np\\n\",\n    \"import tensorflow as tf\\n\",\n    \"from tensorflow.keras import layers\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"id\": \"hollow-environment\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 10 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"class DataGenerator(tf.keras.utils.Sequence):\\n\",\n    \"    def __init__(self, seqs, vocab_sizes, list_IDs, max_visit, max_code, batch_size=100, shuffle=True):\\n\",\n    \"        self.seqs = seqs\\n\",\n    \"        self.code_vocab = vocab_sizes[0]\\n\",\n    \"        self.cat_vocab = vocab_sizes[1]\\n\",\n    \"        self.list_IDs = list_IDs\\n\",\n    \"        self.max_visit = max_visit\\n\",\n    \"        self.max_code = max_code\\n\",\n    \"        self.batch_size = batch_size\\n\",\n    \"        self.shuffle = shuffle\\n\",\n    \"        self.on_epoch_end()\\n\",\n    \"\\n\",\n    \"    def __len__(self):\\n\",\n    \"        'Denotes the number of batches per epoch'\\n\",\n    \"        return int(np.ceil(len(self.list_IDs) / self.batch_size))\\n\",\n    \"\\n\",\n    \"    def __getitem__(self, index):\\n\",\n    \"        'Generate one batch of data'\\n\",\n    \"        indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]\\n\",\n    \"        list_IDs_temp = [self.list_IDs[k] for k in indexes]\\n\",\n    \"        X, y = self.__data_generation(list_IDs_temp)\\n\",\n    \"        return X, y\\n\",\n    \"\\n\",\n    \"    def on_epoch_end(self):\\n\",\n    \"        'Updates indexes after each epoch' \\n\",\n    \"        self.indexes = np.arange(len(self.list_IDs))\\n\",\n    \"        if self.shuffle == True:\\n\",\n    \"            np.random.shuffle(self.indexes)\\n\",\n    \"\\n\",\n    \"    def __data_generation(self, list_IDs_temp):\\n\",\n    \"        'Generates data containing batch_size samples' \\n\",\n    \"        demo_feature, code_feature, util_feature, date_feature, cls_feature = self.seqs\\n\",\n    \"        batch_demo, batch_code, batch_util, batch_date, batch_cls = [], [], [], [], []\\n\",\n    \"        for i, ID in enumerate(list_IDs_temp):\\n\",\n    \"            batch_demo.append(demo_feature[ID])\\n\",\n    \"            batch_code.append(code_feature[ID])\\n\",\n    \"            batch_util.append(util_feature[ID])\\n\",\n    \"            batch_date.append(date_feature[ID])\\n\",\n    \"            batch_cls.append(cls_feature[ID])\\n\",\n    \"        \\n\",\n    \"        batch_demo_feature = np.array(batch_demo)\\n\",\n    \"        batch_code_feature = self.code_padding(batch_code)\\n\",\n    \"        batch_util_feature = self.date_padding(batch_util)\\n\",\n    \"        batch_date_feature = self.date_padding(batch_date)\\n\",\n    \"        batch_cls = np.array(batch_cls)\\n\",\n    \"        \\n\",\n    \"        dic = (\\n\",\n    \"            {\\n\",\n    \"                'demo_feature': batch_demo_feature,\\n\",\n    \"                'code_feature': batch_code_feature,\\n\",\n    \"                'util_feature': batch_util_feature,\\n\",\n    \"                'date_feature': batch_date_feature,\\n\",\n    \"            },\\n\",\n    \"            {\\n\",\n    \"                'cls_label': batch_cls\\n\",\n    \"            })\\n\",\n    \"        return dic\\n\",\n    \"    \\n\",\n    \"    def date_padding(self, seq):\\n\",\n    \"        seq = [x[:-1] for x in seq]\\n\",\n    \"        \\n\",\n    \"        pad_seq = np.zeros((len(seq), self.max_visit))\\n\",\n    \"        for i, p in enumerate(seq):\\n\",\n    \"            pad_seq[i][:len(p)] = p[:self.max_visit]\\n\",\n    \"        return pad_seq\\n\",\n    \"    \\n\",\n    \"    def code_padding(self, seq):\\n\",\n    \"        seq = [x[:-1] for x in seq]\\n\",\n    \"        \\n\",\n    \"        X = np.zeros((len(seq), self.max_visit, self.max_code))\\n\",\n    \"        for i, p in enumerate(seq):\\n\",\n    \"            if len(p) > self.max_visit: \\n\",\n    \"                p = p[:self.max_visit]\\n\",\n    \"            for j, claim in enumerate(p):\\n\",\n    \"                claim = claim[:self.max_code]\\n\",\n    \"                X[i][j][:len(claim)] = claim\\n\",\n    \"        return X\\n\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"id\": \"quick-webster\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 18.3 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"def create_code_mask(code_seq):\\n\",\n    \"    code_mask = tf.cast(tf.math.not_equal(code_seq, 0), tf.float32)\\n\",\n    \"    return code_mask[:,:,:,tf.newaxis]\\n\",\n    \"\\n\",\n    \"def create_visit_mask(seq):\\n\",\n    \"    visit_mask = tf.cast(tf.math.not_equal(seq, 0), tf.float32)\\n\",\n    \"    return visit_mask[:,:]\\n\",\n    \"\\n\",\n    \"def scaled_dot_product_attention(Q, K, V, Q_masks, K_masks):\\n\",\n    \"    d_k = K.get_shape().as_list()[-1] # d_model/h\\n\",\n    \"\\n\",\n    \"    outputs = tf.matmul(Q, tf.transpose(K, [0, 2, 1]))  # (h*N, T_q, T_k)\\n\",\n    \"    outputs /= d_k ** 0.5\\n\",\n    \"\\n\",\n    \"    padding_num = -1e+7\\n\",\n    \"    K_masks = tf.expand_dims(K_masks, 1) # (h*N, 1, T_k)\\n\",\n    \"    K_masks = tf.tile(K_masks, [1, tf.shape(Q)[1], 1]) # (h*N, T_q, T_k)\\n\",\n    \"    paddings = tf.ones_like(outputs) * padding_num\\n\",\n    \"    outputs = tf.where(tf.equal(K_masks, 0), paddings, outputs)  # (h*N, T_q, T_k)\\n\",\n    \"\\n\",\n    \"    outputs = tf.nn.softmax(outputs)\\n\",\n    \"    attention = outputs\\n\",\n    \"    Q_masks = tf.expand_dims(Q_masks, -1) # (h*N, T_q, 1)\\n\",\n    \"    Q_masks = tf.tile(Q_masks, [1, 1, tf.shape(K)[1]]) # (h*N, T_q, T_k)\\n\",\n    \"    outputs = outputs * tf.cast(Q_masks, dtype=tf.float32)\\n\",\n    \"\\n\",\n    \"    return tf.matmul(outputs, V), attention # [h*N, T_q, d_model/h]\\n\",\n    \"\\n\",\n    \"class multihead_attention(tf.keras.layers.Layer):\\n\",\n    \"    def __init__(self, d_model, num_heads, name=\\\"multihead_attention\\\"):\\n\",\n    \"        super(multihead_attention, self).__init__(name=name)\\n\",\n    \"        self.num_heads = num_heads\\n\",\n    \"        self.d_model = d_model\\n\",\n    \"\\n\",\n    \"        assert d_model % self.num_heads == 0\\n\",\n    \"\\n\",\n    \"        self.query_dense = layers.Dense(units=d_model, use_bias=False)\\n\",\n    \"        self.key_dense = layers.Dense(units=d_model, use_bias=False)\\n\",\n    \"        self.value_dense = layers.Dense(units=d_model, use_bias=False)\\n\",\n    \"        self.add =layers.Add()\\n\",\n    \"        self.norm = layers.LayerNormalization()\\n\",\n    \"    \\n\",\n    \"    def call(self, queries, keys, values, query_masks, key_masks):\\n\",\n    \"        Q = self.query_dense(queries)\\n\",\n    \"        K = self.key_dense(keys)\\n\",\n    \"        V = self.value_dense(values)\\n\",\n    \"\\n\",\n    \"        # Split and concat\\n\",\n    \"        Q_ = tf.concat(tf.split(Q, self.num_heads, axis=2), axis=0) # (h*N, T_q, d_model/h)\\n\",\n    \"        K_ = tf.concat(tf.split(K, self.num_heads, axis=2), axis=0) # (h*N, T_k, d_model/h)\\n\",\n    \"        V_ = tf.concat(tf.split(V, self.num_heads, axis=2), axis=0) # (h*N, T_v, d_model/h)\\n\",\n    \"        query_masks = tf.tile(query_masks, [self.num_heads, 1]) # (h*N, T_q)\\n\",\n    \"        key_masks = tf.tile(key_masks, [self.num_heads, 1]) # (h*N, T_k)\\n\",\n    \"\\n\",\n    \"        # Attention\\n\",\n    \"        outputs, attention = scaled_dot_product_attention(Q_, K_, V_, query_masks, key_masks) # (h*N, T_q, d_model/h)\\n\",\n    \"\\n\",\n    \"        # Restore shape\\n\",\n    \"        outputs = tf.concat(tf.split(outputs, self.num_heads, axis=0), axis=2) # (N, T_q, d_model)\\n\",\n    \"\\n\",\n    \"        # Residual connection\\n\",\n    \"        outputs = self.add([queries, outputs])\\n\",\n    \"        outputs = self.norm(outputs)\\n\",\n    \"        \\n\",\n    \"        return outputs, attention\\n\",\n    \"\\n\",\n    \"class ffn(tf.keras.layers.Layer):\\n\",\n    \"    def __init__(self, d_model, ffn_dim, name=\\\"ffn\\\"):\\n\",\n    \"        super(ffn, self).__init__(name=name)\\n\",\n    \"        self.ffn_dim = ffn_dim\\n\",\n    \"        self.dense1 = layers.Dense(units=ffn_dim, activation=tf.nn.relu, use_bias=False)\\n\",\n    \"        self.dense2 = layers.Dense(units=d_model, use_bias=False)\\n\",\n    \"        self.add =layers.Add()\\n\",\n    \"        self.norm = layers.LayerNormalization()\\n\",\n    \"    \\n\",\n    \"    def call(self, inputs):\\n\",\n    \"        outputs = self.dense1(inputs)\\n\",\n    \"        outputs = self.dense2(outputs)\\n\",\n    \"        outputs = self.add([inputs, outputs])\\n\",\n    \"        outputs = self.norm(outputs)\\n\",\n    \"        return outputs\\n\",\n    \"\\n\",\n    \"def cat_recall(y_true, y_pred):\\n\",\n    \"    mask_value = tf.cast(tf.not_equal(tf.reduce_sum(y_true,axis=-1), 0), tf.float32)\\n\",\n    \"    true_positives = tf.cast(tf.reduce_sum(tf.multiply(tf.round(y_pred), y_true), axis=-1), tf.float32)\\n\",\n    \"    possible_positives = tf.cast(tf.reduce_sum(y_true, axis=-1), tf.float32)\\n\",\n    \"    values = true_positives / (possible_positives + 1e-7)\\n\",\n    \"    return tf.reduce_sum(values)/tf.reduce_sum(mask_value)\\n\",\n    \"\\n\",\n    \"def cat_loss_fun(y_true, y_pred):\\n\",\n    \"    loss = tf.cast(tf.keras.losses.BinaryCrossentropy(reduction='none')(y_true, y_pred), tf.float32)\\n\",\n    \"    mask = tf.cast(tf.not_equal(tf.reduce_sum(y_true,axis=-1), 0), tf.float32)\\n\",\n    \"    loss = tf.multiply(loss, mask)\\n\",\n    \"    # return tf.reduce_sum(loss)/tf.reduce_sum(mask)\\n\",\n    \"    return loss\\n\",\n    \"\\n\",\n    \"def model(\\n\",\n    \"    max_visit,\\n\",\n    \"    max_code,\\n\",\n    \"    max_demo,\\n\",\n    \"    \\n\",\n    \"    demo_vocab,\\n\",\n    \"    code_vocab,\\n\",\n    \"    date_vocab,\\n\",\n    \"    util_vocab,\\n\",\n    \"    cat_vocab,\\n\",\n    \"\\n\",\n    \"    patient_dim,\\n\",\n    \"    vocab_dim=100,\\n\",\n    \"    model_dim=100,\\n\",\n    \"    ffn_dim=100,\\n\",\n    \"    num_heads=2,\\n\",\n    \"    num_translayer=1,\\n\",\n    \"    \\n\",\n    \"    model_name=\\\"TransF\\\"):\\n\",\n    \"    \\n\",\n    \"    demo = layers.Input(shape=(max_demo, ), name=\\\"demo_feature\\\")  # max_demo = 2, age&sex\\n\",\n    \"    code_seq = layers.Input(shape=(max_visit, max_code), name=\\\"code_feature\\\") \\n\",\n    \"    util_seq = layers.Input(shape=(max_visit), name=\\\"util_feature\\\")\\n\",\n    \"    date_seq = layers.Input(shape=(max_visit), name=\\\"date_feature\\\")\\n\",\n    \"\\n\",\n    \"    inputs = [demo, code_seq, util_seq, date_seq]\\n\",\n    \"    \\n\",\n    \"    # demo embedding\\n\",\n    \"    demo_emb = layers.Embedding(input_dim=demo_vocab, output_dim=vocab_dim, mask_zero=True, name='demo_embedding')(demo)\\n\",\n    \"    demo_emb = layers.Lambda(lambda x: tf.keras.backend.sum(x, axis=1))(demo_emb)     \\n\",\n    \"\\n\",\n    \"    # code sequence\\n\",\n    \"    code_mask = layers.Lambda(create_code_mask)(code_seq)\\n\",\n    \"    code_emb = layers.Embedding(input_dim=code_vocab, \\n\",\n    \"                                output_dim=vocab_dim, \\n\",\n    \"                                name='code_embed')(code_seq)\\n\",\n    \"    code_emb = layers.Multiply()([code_emb, code_mask])\\n\",\n    \"    code_emb = tf.reduce_sum(code_emb, axis=2)     \\n\",\n    \"\\n\",\n    \"    \\n\",\n    \"    # visit mask\\n\",\n    \"    visit_mask = layers.Lambda(create_visit_mask)(date_seq)\\n\",\n    \"    \\n\",\n    \"    # util sequence   \\n\",\n    \"    util_emb = layers.Embedding(input_dim=util_vocab, output_dim=vocab_dim, mask_zero=True, name='util_embedding')(util_seq)\\n\",\n    \"    util_emb = layers.Multiply()([util_emb, visit_mask[:,:,tf.newaxis]])\\n\",\n    \"    \\n\",\n    \"\\n\",\n    \"    return tf.keras.Model(inputs=inputs, outputs=outputs, name=model_name)\\n\",\n    \"\\n\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 7,\n   \"id\": \"dependent-movie\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 27 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"def process_code(seq, vocab2int):\\n\",\n    \"    unseen = []\\n\",\n    \"    new_seq = []\\n\",\n    \"    for p in seq:\\n\",\n    \"        new_p = []\\n\",\n    \"        for v in p:\\n\",\n    \"            new_v = []\\n\",\n    \"            for c in v:\\n\",\n    \"                if c not in vocab2int: \\n\",\n    \"                    unseen.append(c)\\n\",\n    \"                    continue\\n\",\n    \"                    # vocab2int[c] = len(vocab2int)\\n\",\n    \"                new_v.append(vocab2int[c])\\n\",\n    \"            new_p.append(new_v)\\n\",\n    \"        new_seq.append(new_p)\\n\",\n    \"        \\n\",\n    \"    print(\\\"UNSEEN VOCAB:\\\",len(set(unseen)), len(unseen))\\n\",\n    \"    return new_seq\\n\",\n    \"\\n\",\n    \"def process_util(seq, util2int):\\n\",\n    \"    new_seq = []\\n\",\n    \"    vocab2int = {\\\"PAD\\\":0,\\\"IP\\\":1,\\\"RX\\\":2,\\\"OP\\\":3}\\n\",\n    \"    for p in seq:\\n\",\n    \"        new_p = []\\n\",\n    \"        for v in p:\\n\",\n    \"            if \\\"IP\\\" in v:\\n\",\n    \"                new_v=1\\n\",\n    \"            elif \\\"RX\\\" in v:\\n\",\n    \"                new_v=2\\n\",\n    \"            else:\\n\",\n    \"                new_v=3\\n\",\n    \"            new_p.append(new_v)\\n\",\n    \"        new_seq.append(new_p)\\n\",\n    \"    return new_seq\\n\",\n    \"    \\n\",\n    \"def process_demo(age_seq, sex_seq, vocab2int):\\n\",\n    \"    new_seq = []\\n\",\n    \"    for age, sex in zip(age_seq, sex_seq):\\n\",\n    \"        p = []\\n\",\n    \"        assert age in vocab2int\\n\",\n    \"        assert sex in vocab2int\\n\",\n    \"        \\n\",\n    \"        p.append(vocab2int[age])\\n\",\n    \"        p.append(vocab2int[sex])\\n\",\n    \"        new_seq.append(p)\\n\",\n    \"    return np.array(new_seq)\\n\",\n    \"\\n\",\n    \"def get_cat(seq,code2cat):\\n\",\n    \"    new_seq = []\\n\",\n    \"    for p in seq:\\n\",\n    \"        new_p = []\\n\",\n    \"        for v in p:\\n\",\n    \"            new_v = []\\n\",\n    \"            for c in v:\\n\",\n    \"                new_c = code2cat[c]\\n\",\n    \"                new_v.append(new_c)\\n\",\n    \"            new_p.append(new_v)\\n\",\n    \"        new_seq.append(new_p)\\n\",\n    \"    return new_seq\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 8,\n   \"id\": \"ongoing-manufacturer\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"------LOADING DIC------\\n\",\n      \"time: 89.1 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(\\\"------LOADING DIC------\\\")\\n\",\n    \"path = \\\"/Users/xxz005/Desktop/RAW_DATA/code2cat/\\\"\\n\",\n    \"\\n\",\n    \"diag2cat = pickle.load(open(path+\\\"diag2cat\\\",\\\"rb\\\"))\\n\",\n    \"proc2cat = pickle.load(open(path+\\\"proc2cat\\\",\\\"rb\\\"))\\n\",\n    \"drug2cat = pickle.load(open(path+\\\"drug2cat\\\",\\\"rb\\\"))\\n\",\n    \"\\n\",\n    \"code2cat = {**diag2cat, **proc2cat, **drug2cat}\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 9,\n   \"id\": \"wrong-warrior\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 46.1 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"code2int, util2int, demo2int, cat2int  = pickle.load(open(\\\"../../pretraining/model/vocabs/vocabs\\\",\\\"rb\\\"))\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 10,\n   \"id\": \"invalid-silicon\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"UNSEEN VOCAB: 0 0\\n\",\n      \"time: 11.3 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"code_feature = process_code(code_seq, code2int)\\n\",\n    \"util_feature = process_util(util_seq, util2int)\\n\",\n    \"demo_feature = process_demo(age_seq, sex_seq, demo2int)\\n\",\n    \"date_feature = date_seq\\n\",\n    \"\\n\",\n    \"cls_feature = np.array(label_seq).reshape((-1,1))\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 11,\n   \"id\": \"adverse-arkansas\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 2.21 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"MAX_VISIT=30\\n\",\n    \"MAX_CODE=10\\n\",\n    \"MAX_DEMO=2\\n\",\n    \"PATIENT_DIM=100\\n\",\n    \"\\n\",\n    \"BATCH_SIZE = 500\\n\",\n    \"TRAIN_RATIO = 0.7\\n\",\n    \"DATA_SIZE = len(age_seq)\\n\",\n    \"EPOCHS = 20\\n\",\n    \"\\n\",\n    \"params = {\\n\",\n    \"    'seqs':[demo_feature, code_feature, util_feature, date_feature, cls_feature],\\n\",\n    \"    'vocab_sizes': [len(code2int), len(cat2int)],\\n\",\n    \"    'batch_size':100,\\n\",\n    \"    'max_visit':MAX_VISIT, \\n\",\n    \"    'max_code':MAX_CODE,\\n\",\n    \"}\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 12,\n   \"id\": \"sorted-wholesale\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 1.69 s\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"from sklearn.model_selection import train_test_split\\n\",\n    \"train_IDs, valid_IDs = train_test_split(range(DATA_SIZE), train_size=TRAIN_RATIO, random_state=42)\\n\",\n    \"train_generator = DataGenerator(list_IDs=train_IDs, shuffle=True, **params)\\n\",\n    \"valid_generator = DataGenerator(list_IDs=valid_IDs, shuffle=False, **params)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"separated-layout\",\n   \"metadata\": {},\n   \"source\": [\n    \"# Pretrain\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 13,\n   \"id\": \"suspended-settlement\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 417 µs\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"# model_path = \\\"../../pretraining/model/saveModel\\\"\\n\",\n    \"\\n\",\n    \"# model = tf.keras.models.load_model(model_path)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 15,\n   \"id\": \"handmade-memorabilia\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 1.67 s\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"model_path = \\\"/Users/xxz005/Desktop/saveModel\\\"\\n\",\n    \"\\n\",\n    \"model = tf.keras.models.load_model(model_path)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 16,\n   \"id\": \"involved-genetics\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 7.8 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"model_losses = {\\n\",\n    \"    \\\"cls_label\\\":tf.keras.losses.BinaryCrossentropy(),\\n\",\n    \"}\\n\",\n    \"\\n\",\n    \"model_metrics = {\\n\",\n    \"    \\\"cls_label\\\": tf.keras.metrics.AUC(),\\n\",\n    \"}\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 17,\n   \"id\": \"veterinary-chinese\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Model: \\\"TransF\\\"\\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"Layer (type)                    Output Shape         Param #     Connected to                     \\n\",\n      \"==================================================================================================\\n\",\n      \"demo_feature (InputLayer)       [(None, 2)]          0                                            \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"demo_embedding (Embedding)      (None, 2, 100)       2400        demo_feature[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"lambda (Lambda)                 (None, 100)          0           demo_embedding[0][0]             \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"code_feature (InputLayer)       [(None, 30, 10)]     0                                            \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"date_feature (InputLayer)       [(None, 30)]         0                                            \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_ExpandDims (TensorF (None, 1, 100)       0           lambda[0][0]                     \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"code_embed (Embedding)          (None, 30, 10, 100)  4983600     code_feature[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"lambda_1 (Lambda)               (None, 30, 10, 1)    0           code_feature[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"lambda_2 (Lambda)               (None, 30)           0           date_feature[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"util_feature (InputLayer)       [(None, 30)]         0                                            \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"multiply (Multiply)             (None, 30, 10, 100)  0           code_embed[0][0]                 \\n\",\n      \"                                                                 lambda_1[0][0]                   \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"date_embedding (Embedding)      (None, 30, 100)      36600       date_feature[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_strided_slice_1 (Te (None, 30, 1)        0           lambda_2[0][0]                   \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"util_embedding (Embedding)      (None, 30, 100)      400         util_feature[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_strided_slice (Tens (None, 30, 1)        0           lambda_2[0][0]                   \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_Sum_1 (TensorFlowOp (None, 1)            0           tf_op_layer_ExpandDims[0][0]     \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_Sum (TensorFlowOpLa (None, 30, 100)      0           multiply[0][0]                   \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"multiply_2 (Multiply)           (None, 30, 100)      0           date_embedding[0][0]             \\n\",\n      \"                                                                 tf_op_layer_strided_slice_1[0][0]\\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"multiply_1 (Multiply)           (None, 30, 100)      0           util_embedding[0][0]             \\n\",\n      \"                                                                 tf_op_layer_strided_slice[0][0]  \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_Shape (TensorFlowOp (2,)                 0           tf_op_layer_Sum_1[0][0]          \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"add (Add)                       (None, 30, 100)      0           tf_op_layer_Sum[0][0]            \\n\",\n      \"                                                                 multiply_2[0][0]                 \\n\",\n      \"                                                                 multiply_1[0][0]                 \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_Fill (TensorFlowOpL (None, 1)            0           tf_op_layer_Shape[0][0]          \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_concat (TensorFlowO (None, 31, 100)      0           tf_op_layer_ExpandDims[0][0]     \\n\",\n      \"                                                                 add[0][0]                        \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_concat_4 (TensorFlo (None, 31)           0           tf_op_layer_Fill[0][0]           \\n\",\n      \"                                                                 lambda_2[0][0]                   \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_concat_1 (TensorFlo (None, 31, 100)      0           tf_op_layer_ExpandDims[0][0]     \\n\",\n      \"                                                                 add[0][0]                        \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_concat_3 (TensorFlo (None, 31)           0           tf_op_layer_Fill[0][0]           \\n\",\n      \"                                                                 lambda_2[0][0]                   \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_concat_2 (TensorFlo (None, 31, 100)      0           tf_op_layer_ExpandDims[0][0]     \\n\",\n      \"                                                                 add[0][0]                        \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"multihead_attention-0 (multihea (None, 31, 100)      30200       tf_op_layer_concat[0][0]         \\n\",\n      \"                                                                 tf_op_layer_concat_4[0][0]       \\n\",\n      \"                                                                 tf_op_layer_concat_1[0][0]       \\n\",\n      \"                                                                 tf_op_layer_concat_3[0][0]       \\n\",\n      \"                                                                 tf_op_layer_concat_2[0][0]       \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"ffn-0 (ffn)                     (None, 31, 100)      20200       multihead_attention-0[0][0]      \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_strided_slice_2 (Te (None, 1, 100)       0           ffn-0[0][0]                      \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_Squeeze (TensorFlow (None, 100)          0           tf_op_layer_strided_slice_2[0][0]\\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"patient_embedding (Dense)       (None, 100)          10100       tf_op_layer_Squeeze[0][0]        \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf_op_layer_strided_slice_3 (Te (None, 30, 100)      0           ffn-0[0][0]                      \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"dense_5 (Dense)                 (None, 100)          10100       patient_embedding[0][0]          \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"code_label (Dense)              (None, 3357)         339057      patient_embedding[0][0]          \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"cat_label (Dense)               (None, 30, 3357)     339057      tf_op_layer_strided_slice_3[0][0]\\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"cls_label (Dense)               (None, 1)            101         dense_5[0][0]                    \\n\",\n      \"==================================================================================================\\n\",\n      \"Total params: 5,771,815\\n\",\n      \"Trainable params: 5,771,815\\n\",\n      \"Non-trainable params: 0\\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"None\\n\",\n      \"time: 34.3 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"opt = tf.keras.optimizers.Adam(learning_rate=0.0001)\\n\",\n    \"model.compile(optimizer=opt, loss=model_losses, metrics=model_metrics)\\n\",\n    \"print(model.summary())\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 18,\n   \"id\": \"vanilla-karen\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"2/2 [==============================] - 1s 25ms/step - loss: 5.3384 - cls_label_loss: 5.3384 - cls_label_auc: 0.5191\\n\"\n     ]\n    },\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"[5.3545451164245605, 5.3545451164245605, 0.5202370285987854]\"\n      ]\n     },\n     \"execution_count\": 18,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    },\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 1.48 s\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"model.evaluate(valid_generator)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 19,\n   \"id\": \"verbal-kingdom\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Epoch 1/20\\n\",\n      \"WARNING:tensorflow:Gradients do not exist for variables ['code_label/kernel:0', 'code_label/bias:0', 'cat_label/kernel:0', 'cat_label/bias:0'] when minimizing the loss.\\n\",\n      \"WARNING:tensorflow:Gradients do not exist for variables ['code_label/kernel:0', 'code_label/bias:0', 'cat_label/kernel:0', 'cat_label/bias:0'] when minimizing the loss.\\n\",\n      \"4/4 - 2s - loss: 2.8303 - cls_label_loss: 2.8303 - cls_label_auc: 0.5014 - val_loss: 0.8291 - val_cls_label_loss: 0.8291 - val_cls_label_auc: 0.5882\\n\",\n      \"Epoch 2/20\\n\",\n      \"4/4 - 0s - loss: 1.3828 - cls_label_loss: 1.3828 - cls_label_auc: 0.4872 - val_loss: 1.3247 - val_cls_label_loss: 1.3247 - val_cls_label_auc: 0.5479\\n\",\n      \"Epoch 3/20\\n\",\n      \"4/4 - 0s - loss: 1.4830 - cls_label_loss: 1.4830 - cls_label_auc: 0.5295 - val_loss: 0.9591 - val_cls_label_loss: 0.9591 - val_cls_label_auc: 0.5396\\n\",\n      \"Epoch 4/20\\n\",\n      \"4/4 - 0s - loss: 0.9104 - cls_label_loss: 0.9104 - cls_label_auc: 0.5244 - val_loss: 0.8159 - val_cls_label_loss: 0.8159 - val_cls_label_auc: 0.5259\\n\",\n      \"Epoch 5/20\\n\",\n      \"4/4 - 0s - loss: 0.8908 - cls_label_loss: 0.8908 - cls_label_auc: 0.5669 - val_loss: 1.0094 - val_cls_label_loss: 1.0094 - val_cls_label_auc: 0.5249\\n\",\n      \"Epoch 6/20\\n\",\n      \"4/4 - 0s - loss: 0.7480 - cls_label_loss: 0.7480 - cls_label_auc: 0.6431 - val_loss: 0.6296 - val_cls_label_loss: 0.6296 - val_cls_label_auc: 0.5945\\n\",\n      \"Epoch 7/20\\n\",\n      \"4/4 - 0s - loss: 0.6394 - cls_label_loss: 0.6394 - cls_label_auc: 0.6756 - val_loss: 0.6210 - val_cls_label_loss: 0.6210 - val_cls_label_auc: 0.6494\\n\",\n      \"Epoch 8/20\\n\",\n      \"4/4 - 0s - loss: 0.6747 - cls_label_loss: 0.6747 - cls_label_auc: 0.6974 - val_loss: 0.5902 - val_cls_label_loss: 0.5902 - val_cls_label_auc: 0.6696\\n\",\n      \"Epoch 9/20\\n\",\n      \"4/4 - 0s - loss: 0.5923 - cls_label_loss: 0.5923 - cls_label_auc: 0.7106 - val_loss: 0.6152 - val_cls_label_loss: 0.6152 - val_cls_label_auc: 0.6576\\n\",\n      \"Epoch 10/20\\n\",\n      \"4/4 - 0s - loss: 0.5903 - cls_label_loss: 0.5903 - cls_label_auc: 0.7281 - val_loss: 0.6278 - val_cls_label_loss: 0.6278 - val_cls_label_auc: 0.6683\\n\",\n      \"Epoch 11/20\\n\",\n      \"4/4 - 0s - loss: 0.5430 - cls_label_loss: 0.5430 - cls_label_auc: 0.7677 - val_loss: 0.5627 - val_cls_label_loss: 0.5627 - val_cls_label_auc: 0.6906\\n\",\n      \"Epoch 12/20\\n\",\n      \"4/4 - 0s - loss: 0.5206 - cls_label_loss: 0.5206 - cls_label_auc: 0.7955 - val_loss: 0.5575 - val_cls_label_loss: 0.5575 - val_cls_label_auc: 0.6949\\n\",\n      \"Epoch 13/20\\n\",\n      \"4/4 - 0s - loss: 0.4970 - cls_label_loss: 0.4970 - cls_label_auc: 0.8233 - val_loss: 0.5646 - val_cls_label_loss: 0.5646 - val_cls_label_auc: 0.6961\\n\",\n      \"Epoch 14/20\\n\",\n      \"4/4 - 0s - loss: 0.4812 - cls_label_loss: 0.4812 - cls_label_auc: 0.8363 - val_loss: 0.5748 - val_cls_label_loss: 0.5748 - val_cls_label_auc: 0.7066\\n\",\n      \"Epoch 15/20\\n\",\n      \"4/4 - 0s - loss: 0.4676 - cls_label_loss: 0.4676 - cls_label_auc: 0.8548 - val_loss: 0.5503 - val_cls_label_loss: 0.5503 - val_cls_label_auc: 0.7186\\n\",\n      \"Epoch 16/20\\n\",\n      \"4/4 - 1s - loss: 0.4579 - cls_label_loss: 0.4579 - cls_label_auc: 0.8611 - val_loss: 0.5309 - val_cls_label_loss: 0.5309 - val_cls_label_auc: 0.7301\\n\",\n      \"Epoch 17/20\\n\",\n      \"4/4 - 0s - loss: 0.4415 - cls_label_loss: 0.4415 - cls_label_auc: 0.8793 - val_loss: 0.5320 - val_cls_label_loss: 0.5320 - val_cls_label_auc: 0.7366\\n\",\n      \"Epoch 18/20\\n\",\n      \"4/4 - 0s - loss: 0.4196 - cls_label_loss: 0.4196 - cls_label_auc: 0.8958 - val_loss: 0.5427 - val_cls_label_loss: 0.5427 - val_cls_label_auc: 0.7388\\n\",\n      \"Epoch 19/20\\n\",\n      \"4/4 - 0s - loss: 0.4135 - cls_label_loss: 0.4135 - cls_label_auc: 0.9036 - val_loss: 0.5512 - val_cls_label_loss: 0.5512 - val_cls_label_auc: 0.7338\\n\",\n      \"Epoch 20/20\\n\",\n      \"4/4 - 0s - loss: 0.3943 - cls_label_loss: 0.3943 - cls_label_auc: 0.9157 - val_loss: 0.5306 - val_cls_label_loss: 0.5306 - val_cls_label_auc: 0.7374\\n\",\n      \"time: 11.1 s\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"finetune_history = model.fit(\\n\",\n    \"    train_generator,\\n\",\n    \"    epochs=EPOCHS,\\n\",\n    \"    validation_data=valid_generator,\\n\",\n    \"    verbose=2,\\n\",\n    \")\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 20,\n   \"id\": \"conventional-makeup\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 442 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"outs = model.predict(valid_generator)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 21,\n   \"id\": \"logical-opinion\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"((147, 3357), (147, 30, 3357), (147, 1))\"\n      ]\n     },\n     \"execution_count\": 21,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    },\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 3.12 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"outs[0].shape, outs[1].shape, outs[2].shape\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"hundred-idaho\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"banner-incentive\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"median-beads\",\n   \"metadata\": {},\n   \"source\": [\n    \"# Cold Start\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 144,\n   \"id\": \"particular-corps\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 512 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"m = model(\\n\",\n    \"    patient_dim=PATIENT_DIM,\\n\",\n    \"    max_visit=MAX_VISIT,\\n\",\n    \"    max_code=MAX_CODE,\\n\",\n    \"    max_demo=MAX_DEMO,\\n\",\n    \"    code_vocab=len(code2int),\\n\",\n    \"    demo_vocab=len(demo2int),\\n\",\n    \"    util_vocab=4,\\n\",\n    \"    date_vocab=365,\\n\",\n    \"    cat_vocab=len(cat2int),\\n\",\n    \")\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"demonstrated-congress\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 146,\n   \"id\": \"frozen-borough\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Model: \\\"TransF\\\"\\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"Layer (type)                    Output Shape         Param #     Connected to                     \\n\",\n      \"==================================================================================================\\n\",\n      \"demo_feature (InputLayer)       [(None, 2)]          0                                            \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"demo_embedding (Embedding)      (None, 2, 100)       2400        demo_feature[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"code_feature (InputLayer)       [(None, 30, 10)]     0                                            \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"date_feature (InputLayer)       [(None, 30)]         0                                            \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"lambda_9 (Lambda)               (None, 100)          0           demo_embedding[0][0]             \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"code_embed (Embedding)          (None, 30, 10, 100)  4983600     code_feature[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"lambda_10 (Lambda)              (None, 30, 10, 1)    0           code_feature[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"lambda_11 (Lambda)              (None, 30)           0           date_feature[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"util_feature (InputLayer)       [(None, 30)]         0                                            \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.expand_dims_3 (TFOpLambda)   (None, 1, 100)       0           lambda_9[0][0]                   \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"multiply_9 (Multiply)           (None, 30, 10, 100)  0           code_embed[0][0]                 \\n\",\n      \"                                                                 lambda_10[0][0]                  \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"date_embedding (Embedding)      (None, 30, 100)      36500       date_feature[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.__operators__.getitem_13 (Sl (None, 30, 1)        0           lambda_11[0][0]                  \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"util_embedding (Embedding)      (None, 30, 100)      400         util_feature[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.__operators__.getitem_12 (Sl (None, 30, 1)        0           lambda_11[0][0]                  \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.math.reduce_sum_6 (TFOpLambd (None, 30, 100)      0           multiply_9[0][0]                 \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"multiply_11 (Multiply)          (None, 30, 100)      0           date_embedding[0][0]             \\n\",\n      \"                                                                 tf.__operators__.getitem_13[0][0]\\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"multiply_10 (Multiply)          (None, 30, 100)      0           util_embedding[0][0]             \\n\",\n      \"                                                                 tf.__operators__.getitem_12[0][0]\\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.math.reduce_sum_7 (TFOpLambd (None, 1)            0           tf.expand_dims_3[0][0]           \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"add_9 (Add)                     (None, 30, 100)      0           tf.math.reduce_sum_6[0][0]       \\n\",\n      \"                                                                 multiply_11[0][0]                \\n\",\n      \"                                                                 multiply_10[0][0]                \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.ones_like_3 (TFOpLambda)     (None, 1)            0           tf.math.reduce_sum_7[0][0]       \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.concat_15 (TFOpLambda)       (None, 31, 100)      0           tf.expand_dims_3[0][0]           \\n\",\n      \"                                                                 add_9[0][0]                      \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.concat_19 (TFOpLambda)       (None, 31)           0           tf.ones_like_3[0][0]             \\n\",\n      \"                                                                 lambda_11[0][0]                  \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.concat_16 (TFOpLambda)       (None, 31, 100)      0           tf.expand_dims_3[0][0]           \\n\",\n      \"                                                                 add_9[0][0]                      \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.concat_18 (TFOpLambda)       (None, 31)           0           tf.ones_like_3[0][0]             \\n\",\n      \"                                                                 lambda_11[0][0]                  \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.concat_17 (TFOpLambda)       (None, 31, 100)      0           tf.expand_dims_3[0][0]           \\n\",\n      \"                                                                 add_9[0][0]                      \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"multihead_attention-0 (multihea (None, 31, 100)      30200       tf.concat_15[0][0]               \\n\",\n      \"                                                                 tf.concat_19[0][0]               \\n\",\n      \"                                                                 tf.concat_16[0][0]               \\n\",\n      \"                                                                 tf.concat_18[0][0]               \\n\",\n      \"                                                                 tf.concat_17[0][0]               \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"ffn-0 (ffn)                     (None, 31, 100)      20200       multihead_attention-0[0][0]      \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.__operators__.getitem_14 (Sl (None, 1, 100)       0           ffn-0[0][0]                      \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"tf.compat.v1.squeeze_3 (TFOpLam (None, 100)          0           tf.__operators__.getitem_14[0][0]\\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"patient_embedding (Dense)       (None, 100)          10100       tf.compat.v1.squeeze_3[0][0]     \\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"code_label (Dense)              (None, 3357)         339057      patient_embedding[0][0]          \\n\",\n      \"==================================================================================================\\n\",\n      \"Total params: 5,422,457\\n\",\n      \"Trainable params: 5,422,457\\n\",\n      \"Non-trainable params: 0\\n\",\n      \"__________________________________________________________________________________________________\\n\",\n      \"None\\n\",\n      \"time: 33.7 ms\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"m.compile(optimizer=\\\"RMSprop\\\", loss=tf.keras.losses.BinaryCrossentropy(), metrics=[tf.keras.metrics.Recall(top_k=5), \\n\",\n    \"                                                                                       tf.keras.metrics.Recall(top_k=10), \\n\",\n    \"                                                                                       tf.keras.metrics.Recall(top_k=30)])\\n\",\n    \"print(m.summary())\\n\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 147,\n   \"id\": \"structural-space\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"time: 325 µs\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"# opt = tf.keras.optimizers.Adam(learning_rate=0.01)\\n\",\n    \"# m.compile(optimizer=opt, loss=tf.keras.losses.BinaryCrossentropy(), metrics=[tf.keras.metrics.Recall(top_k=5), \\n\",\n    \"#                                                                                        tf.keras.metrics.Recall(top_k=10), \\n\",\n    \"#                                                                                        tf.keras.metrics.Recall(top_k=30)])\\n\",\n    \"# print(m.summary())\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 148,\n   \"id\": \"cleared-swimming\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Epoch 1/20\\n\",\n      \"70/70 - 11s - loss: 0.0524 - recall_41: 0.1668 - recall_42: 0.2301 - recall_43: 0.3743 - val_loss: 0.0047 - val_recall_41: 0.1927 - val_recall_42: 0.2697 - val_recall_43: 0.4610\\n\",\n      \"Epoch 2/20\\n\",\n      \"70/70 - 9s - loss: 0.0045 - recall_41: 0.1983 - recall_42: 0.2773 - recall_43: 0.4633 - val_loss: 0.0045 - val_recall_41: 0.1966 - val_recall_42: 0.2817 - val_recall_43: 0.4795\\n\",\n      \"Epoch 3/20\\n\",\n      \"70/70 - 9s - loss: 0.0044 - recall_41: 0.2138 - recall_42: 0.3049 - recall_43: 0.4900 - val_loss: 0.0044 - val_recall_41: 0.2262 - val_recall_42: 0.3118 - val_recall_43: 0.5048\\n\",\n      \"Epoch 4/20\\n\",\n      \"70/70 - 9s - loss: 0.0043 - recall_41: 0.2468 - recall_42: 0.3443 - recall_43: 0.5371 - val_loss: 0.0043 - val_recall_41: 0.2426 - val_recall_42: 0.3308 - val_recall_43: 0.5269\\n\",\n      \"Epoch 5/20\\n\",\n      \"70/70 - 9s - loss: 0.0041 - recall_41: 0.2795 - recall_42: 0.3847 - recall_43: 0.5798 - val_loss: 0.0043 - val_recall_41: 0.2471 - val_recall_42: 0.3388 - val_recall_43: 0.5346\\n\",\n      \"Epoch 6/20\\n\",\n      \"70/70 - 9s - loss: 0.0039 - recall_41: 0.3166 - recall_42: 0.4276 - recall_43: 0.6244 - val_loss: 0.0043 - val_recall_41: 0.2561 - val_recall_42: 0.3418 - val_recall_43: 0.5380\\n\",\n      \"Epoch 7/20\\n\",\n      \"70/70 - 9s - loss: 0.0037 - recall_41: 0.3552 - recall_42: 0.4698 - recall_43: 0.6668 - val_loss: 0.0044 - val_recall_41: 0.2526 - val_recall_42: 0.3348 - val_recall_43: 0.5261\\n\",\n      \"Epoch 8/20\\n\",\n      \"70/70 - 10s - loss: 0.0035 - recall_41: 0.3911 - recall_42: 0.5110 - recall_43: 0.7065 - val_loss: 0.0045 - val_recall_41: 0.2426 - val_recall_42: 0.3263 - val_recall_43: 0.5186\\n\",\n      \"Epoch 9/20\\n\",\n      \"70/70 - 10s - loss: 0.0033 - recall_41: 0.4259 - recall_42: 0.5546 - recall_43: 0.7375 - val_loss: 0.0046 - val_recall_41: 0.2367 - val_recall_42: 0.3212 - val_recall_43: 0.4980\\n\",\n      \"Epoch 10/20\\n\",\n      \"70/70 - 9s - loss: 0.0031 - recall_41: 0.4652 - recall_42: 0.5950 - recall_43: 0.7707 - val_loss: 0.0048 - val_recall_41: 0.2311 - val_recall_42: 0.3182 - val_recall_43: 0.5070\\n\",\n      \"Epoch 11/20\\n\",\n      \"70/70 - 9s - loss: 0.0029 - recall_41: 0.5005 - recall_42: 0.6295 - recall_43: 0.8009 - val_loss: 0.0049 - val_recall_41: 0.2246 - val_recall_42: 0.3073 - val_recall_43: 0.4877\\n\",\n      \"Epoch 12/20\\n\",\n      \"70/70 - 9s - loss: 0.0027 - recall_41: 0.5412 - recall_42: 0.6672 - recall_43: 0.8270 - val_loss: 0.0052 - val_recall_41: 0.2148 - val_recall_42: 0.3006 - val_recall_43: 0.4708\\n\",\n      \"Epoch 13/20\\n\",\n      \"70/70 - 9s - loss: 0.0025 - recall_41: 0.5758 - recall_42: 0.7015 - recall_43: 0.8495 - val_loss: 0.0054 - val_recall_41: 0.2174 - val_recall_42: 0.2999 - val_recall_43: 0.4682\\n\",\n      \"Epoch 14/20\\n\",\n      \"70/70 - 9s - loss: 0.0024 - recall_41: 0.6045 - recall_42: 0.7298 - recall_43: 0.8718 - val_loss: 0.0056 - val_recall_41: 0.1997 - val_recall_42: 0.2814 - val_recall_43: 0.4534\\n\",\n      \"Epoch 15/20\\n\",\n      \"70/70 - 9s - loss: 0.0022 - recall_41: 0.6374 - recall_42: 0.7570 - recall_43: 0.8873 - val_loss: 0.0058 - val_recall_41: 0.2096 - val_recall_42: 0.2888 - val_recall_43: 0.4469\\n\",\n      \"Epoch 16/20\\n\",\n      \"70/70 - 9s - loss: 0.0021 - recall_41: 0.6599 - recall_42: 0.7797 - recall_43: 0.9020 - val_loss: 0.0059 - val_recall_41: 0.2007 - val_recall_42: 0.2748 - val_recall_43: 0.4368\\n\",\n      \"Epoch 17/20\\n\",\n      \"70/70 - 9s - loss: 0.0019 - recall_41: 0.6871 - recall_42: 0.8025 - recall_43: 0.9161 - val_loss: 0.0063 - val_recall_41: 0.2057 - val_recall_42: 0.2788 - val_recall_43: 0.4330\\n\",\n      \"Epoch 18/20\\n\",\n      \"70/70 - 10s - loss: 0.0018 - recall_41: 0.7072 - recall_42: 0.8206 - recall_43: 0.9282 - val_loss: 0.0064 - val_recall_41: 0.1902 - val_recall_42: 0.2625 - val_recall_43: 0.4166\\n\",\n      \"Epoch 19/20\\n\",\n      \"70/70 - 9s - loss: 0.0017 - recall_41: 0.7264 - recall_42: 0.8364 - recall_43: 0.9356 - val_loss: 0.0066 - val_recall_41: 0.1876 - val_recall_42: 0.2670 - val_recall_43: 0.4220\\n\",\n      \"Epoch 20/20\\n\",\n      \"70/70 - 9s - loss: 0.0016 - recall_41: 0.7448 - recall_42: 0.8537 - recall_43: 0.9429 - val_loss: 0.0068 - val_recall_41: 0.1791 - val_recall_42: 0.2466 - val_recall_43: 0.4012\\n\",\n      \"time: 3min 7s\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"cold_start_his = m.fit(\\n\",\n    \"    train_generator,\\n\",\n    \"    epochs=EPOCHS,\\n\",\n    \"    validation_data=valid_generator,\\n\",\n    \"    verbose=2,\\n\",\n    \")\\n\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"quick-license\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"apparent-percentage\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"shaped-variable\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  }\n ],\n \"metadata\": {\n  \"kernelspec\": {\n   \"display_name\": \"Python 3\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.7.11\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n"
  },
  {
    "path": "pretraining/DataGenerator.py",
    "content": "import tensorflow as tf\nimport pandas as pd\nimport numpy as np\n\nclass DataGenerator(tf.keras.utils.Sequence):\n    def __init__(self, seqs, vocab_sizes, list_IDs, max_visit, max_code, batch_size=100, shuffle=True):\n        self.seqs = seqs\n        self.code_vocab = vocab_sizes[0]\n        self.cat_vocab = vocab_sizes[1]\n        self.list_IDs = list_IDs\n        self.max_visit = max_visit\n        self.max_code = max_code\n        self.batch_size = batch_size\n        self.shuffle = shuffle\n        self.on_epoch_end()\n\n    def __len__(self):\n        'Denotes the number of batches per epoch'\n        return int(np.ceil(len(self.list_IDs) / self.batch_size))\n\n    def __getitem__(self, index):\n        'Generate one batch of data'\n        indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]\n        list_IDs_temp = [self.list_IDs[k] for k in indexes]\n        X, y = self.__data_generation(list_IDs_temp)\n        return X, y\n\n    def on_epoch_end(self):\n        'Updates indexes after each epoch' \n        self.indexes = np.arange(len(self.list_IDs))\n        if self.shuffle == True:\n            np.random.shuffle(self.indexes)\n\n    def __data_generation(self, list_IDs_temp):\n        'Generates data containing batch_size samples' \n        demo_feature, code_feature, util_feature, date_feature, cat_feature = self.seqs\n        batch_demo, batch_code, batch_util, batch_date, batch_cat = [], [], [], [], [] \n        for i, ID in enumerate(list_IDs_temp):\n            batch_demo.append(demo_feature[ID])\n            batch_code.append(code_feature[ID])\n            batch_util.append(util_feature[ID])\n            batch_date.append(date_feature[ID])\n            batch_cat.append(cat_feature[ID])\n        \n        batch_demo_feature = np.array(batch_demo)\n        batch_code_feature = self.code_padding(batch_code)\n        batch_util_feature = self.date_padding(batch_util)\n        batch_date_feature = self.date_padding(batch_date)\n        \n        batch_cat_label = self.cat_labelling(batch_cat)\n        batch_code_label = self.code_labelling(batch_cat)  # predict cat instead\n        \n        dic = (\n            {\n                'demo_feature': batch_demo_feature,\n                'code_feature': batch_code_feature,\n                'util_feature': batch_util_feature,\n                'date_feature': batch_date_feature,\n            },\n            {\n                'code_label': batch_code_label,\n                'cat_label': batch_cat_label,\n                'cls_label': np.zeros(batch_cat_label.shape)\n            })\n        return dic\n    \n    def date_padding(self, seq):\n        seq = [x[:-1] for x in seq]\n        \n        pad_seq = np.zeros((len(seq), self.max_visit))\n        for i, p in enumerate(seq):\n            pad_seq[i][:len(p)] = p[:self.max_visit]\n        return pad_seq\n    \n    def code_padding(self, seq):\n        seq = [x[:-1] for x in seq]\n        \n        X = np.zeros((len(seq), self.max_visit, self.max_code))\n        for i, p in enumerate(seq):\n            if len(p) > self.max_visit: \n                p = p[:self.max_visit]\n            for j, claim in enumerate(p):\n                claim = claim[:self.max_code]\n                X[i][j][:len(claim)] = claim\n        return X\n    \n    def code_labelling(self, seq):\n        seq = [x[-1] for x in seq]\n        \n        X = np.zeros((len(seq), self.cat_vocab))\n        for i, claim in enumerate(seq):\n            for c in claim:\n                X[i][c] = 1\n        return X\n\n    def cat_labelling(self, seq):\n        seq = [x[:-1] for x in seq]\n        \n        X = np.zeros((len(seq), self.max_visit, self.cat_vocab))\n        for i, p in enumerate(seq):\n            if len(p) > self.max_visit: \n                p = p[:self.max_visit]\n            for j, claim in enumerate(p):\n                for c in claim:\n                    X[i][j][c] = 1\n        return X\n\ndef process_code(seq, PAD=True):\n    new_seq = []\n    if PAD: vocab2int = {\"PAD\":0}\n    else: vocab2int = {}\n    for p in seq:\n        new_p = []\n        for v in p:\n            new_v = []\n            for c in v:\n                if c not in vocab2int: vocab2int[c] = len(vocab2int)\n                new_v.append(vocab2int[c])\n            new_p.append(new_v)\n        new_seq.append(new_p)\n    return new_seq, vocab2int\n\ndef process_util(seq):\n    new_seq = []\n    vocab2int = {\"PAD\":0,\"IP\":1,\"RX\":2,\"OP\":3}\n    for p in seq:\n        new_p = []\n        for v in p:\n            if \"IP\" in v:\n                new_v=1\n            elif \"RX\" in v:\n                new_v=2\n            else:\n                new_v=3\n            new_p.append(new_v)\n        new_seq.append(new_p)\n    return new_seq, vocab2int\n    \ndef process_demo(age_seq, sex_seq):\n    new_seq = []\n    vocab2int = {}\n    for age, sex in zip(age_seq, sex_seq):\n        p = []\n        if age not in vocab2int: vocab2int[age] = len(vocab2int)\n        if sex not in vocab2int: vocab2int[sex] = len(vocab2int)\n        p.append(vocab2int[age])\n        p.append(vocab2int[sex])\n        new_seq.append(p)\n    return np.array(new_seq), vocab2int\n\ndef get_cat(seq, diag2cat, proc2cat, drug2cat):\n    new_seq = []\n    for p in seq:\n        new_p = []\n        for v in p:\n            new_v = []\n            for c in v:\n                if c in diag2cat :\n                    new_c = diag2cat[c]\n                elif c in proc2cat:\n                    new_c = proc2cat[c]\n                elif c in drug2cat:\n                    new_c = drug2cat[c]\n                else:\n                    new_c = c[:5]\n                new_v.append(new_c)\n            new_p.append(new_v)\n        new_seq.append(new_p)\n    return new_seq\n\n\n\n\n"
  },
  {
    "path": "pretraining/cpt.py",
    "content": "import numpy as np\nimport tensorflow as tf\n\nfrom tensorflow.keras import layers\n\ndef create_code_mask(code_seq):\n    code_mask = tf.cast(tf.math.not_equal(code_seq, 0), tf.float32)\n    return code_mask[:,:,:,tf.newaxis]\n\ndef create_visit_mask(seq):\n    visit_mask = tf.cast(tf.math.not_equal(seq, 0), tf.float32)\n    return visit_mask[:,:]\n\ndef scaled_dot_product_attention(Q, K, V, Q_masks, K_masks):\n    d_k = K.get_shape().as_list()[-1] # d_model/h\n\n    outputs = tf.matmul(Q, tf.transpose(K, [0, 2, 1]))  # (h*N, T_q, T_k)\n    outputs /= d_k ** 0.5\n\n    padding_num = -1e+7\n    K_masks = tf.expand_dims(K_masks, 1) # (h*N, 1, T_k)\n    K_masks = tf.tile(K_masks, [1, tf.shape(Q)[1], 1]) # (h*N, T_q, T_k)\n    paddings = tf.ones_like(outputs) * padding_num\n    outputs = tf.where(tf.equal(K_masks, 0), paddings, outputs)  # (h*N, T_q, T_k)\n\n    outputs = tf.nn.softmax(outputs)\n    Q_masks = tf.expand_dims(Q_masks, -1) # (h*N, T_q, 1)\n    Q_masks = tf.tile(Q_masks, [1, 1, tf.shape(K)[1]]) # (h*N, T_q, T_k)\n    outputs = outputs * tf.cast(Q_masks, dtype=tf.float32)\n\n    return tf.matmul(outputs, V) # [h*N, T_q, d_model/h]\n\nclass multihead_attention(tf.keras.layers.Layer):\n    def __init__(self, d_model, num_heads, name=\"multihead_attention\"):\n        super(multihead_attention, self).__init__(name=name)\n        self.num_heads = num_heads\n        self.d_model = d_model\n\n        assert d_model % self.num_heads == 0\n\n        self.query_dense = layers.Dense(units=d_model, use_bias=False)\n        self.key_dense = layers.Dense(units=d_model, use_bias=False)\n        self.value_dense = layers.Dense(units=d_model, use_bias=False)\n        self.add =layers.Add()\n        self.norm = layers.LayerNormalization()\n    \n    def call(self, queries, keys, values, query_masks, key_masks):\n        Q = self.query_dense(queries)\n        K = self.key_dense(keys)\n        V = self.value_dense(values)\n\n        # Split and concat\n        Q_ = tf.concat(tf.split(Q, self.num_heads, axis=2), axis=0) # (h*N, T_q, d_model/h)\n        K_ = tf.concat(tf.split(K, self.num_heads, axis=2), axis=0) # (h*N, T_k, d_model/h)\n        V_ = tf.concat(tf.split(V, self.num_heads, axis=2), axis=0) # (h*N, T_v, d_model/h)\n        query_masks = tf.tile(query_masks, [self.num_heads, 1]) # (h*N, T_q)\n        key_masks = tf.tile(key_masks, [self.num_heads, 1]) # (h*N, T_k)\n\n        # Attention\n        outputs = scaled_dot_product_attention(Q_, K_, V_, query_masks, key_masks) # (h*N, T_q, d_model/h)\n\n        # Restore shape\n        outputs = tf.concat(tf.split(outputs, self.num_heads, axis=0), axis=2) # (N, T_q, d_model)\n\n        # Residual connection\n        outputs = self.add([queries, outputs])\n        outputs = self.norm(outputs)\n        \n        return outputs\n\nclass ffn(tf.keras.layers.Layer):\n    def __init__(self, d_model, ffn_dim, name=\"ffn\"):\n        super(ffn, self).__init__(name=name)\n        self.ffn_dim = ffn_dim\n        self.dense1 = layers.Dense(units=ffn_dim, activation=tf.nn.relu, use_bias=False)\n        self.dense2 = layers.Dense(units=d_model, use_bias=False)\n        self.add =layers.Add()\n        self.norm = layers.LayerNormalization()\n    \n    def call(self, inputs):\n        outputs = self.dense1(inputs)\n        outputs = self.dense2(outputs)\n        outputs = self.add([inputs, outputs])\n        outputs = self.norm(outputs)\n        return outputs\n\ndef cat_recall(y_true, y_pred):\n    mask_value = tf.cast(tf.not_equal(tf.reduce_sum(y_true,axis=-1), 0), tf.float32)\n    true_positives = tf.cast(tf.reduce_sum(tf.multiply(tf.round(y_pred), y_true), axis=-1), tf.float32)\n    possible_positives = tf.cast(tf.reduce_sum(y_true, axis=-1), tf.float32)\n    values = true_positives / (possible_positives + 1e-7)\n    return tf.reduce_sum(values)/tf.reduce_sum(mask_value)\n\ndef cat_loss_fun(y_true, y_pred):\n    loss = tf.cast(tf.keras.losses.BinaryCrossentropy(reduction='none')(y_true, y_pred), tf.float32)\n    mask = tf.cast(tf.not_equal(tf.reduce_sum(y_true,axis=-1), 0), tf.float32)\n    loss = tf.multiply(loss, mask)\n    # return tf.reduce_sum(loss)/tf.reduce_sum(mask)\n    return loss\n\ndef model(\n    max_visit,\n    max_code,\n    max_demo,\n    \n    demo_vocab,\n    code_vocab,\n    date_vocab,\n    util_vocab,\n    cat_vocab,\n\n    patient_dim,\n    vocab_dim=100,\n    model_dim=100,\n    ffn_dim=100,\n    num_heads=2,\n    num_translayer=1,\n    \n    model_name=\"TransF\"):\n    \n    demo = layers.Input(shape=(max_demo, ), name=\"demo_feature\")  # max_demo = 2, age&sex\n    code_seq = layers.Input(shape=(max_visit, max_code), name=\"code_feature\") \n    util_seq = layers.Input(shape=(max_visit), name=\"util_feature\")\n    date_seq = layers.Input(shape=(max_visit), name=\"date_feature\")\n\n    inputs = [demo, code_seq, util_seq, date_seq]\n    \n    # demo embedding\n    demo_emb = layers.Embedding(input_dim=demo_vocab, output_dim=vocab_dim, mask_zero=True, name='demo_embedding')(demo)\n    demo_emb = layers.Lambda(lambda x: tf.keras.backend.sum(x, axis=1))(demo_emb)     \n\n    # code sequence\n    code_mask = layers.Lambda(create_code_mask)(code_seq)\n    code_emb = layers.Embedding(input_dim=code_vocab, \n                                output_dim=vocab_dim, \n                                name='code_embed')(code_seq)\n    code_emb = layers.Multiply()([code_emb, code_mask])\n    code_emb = tf.reduce_sum(code_emb, axis=2) \n    #code_emb = tf.reduce_mean(code_emb, axis=2) \n    #code_emb = tf.keras.layers.MaxPool1D(code_emb, pool_size=2, strides=1, padding='valid')\n    \n    \n    # visit mask\n    visit_mask = layers.Lambda(create_visit_mask)(date_seq)\n    \n    # util sequence   \n    util_emb = layers.Embedding(input_dim=util_vocab, output_dim=vocab_dim, mask_zero=True, name='util_embedding')(util_seq)\n    util_emb = layers.Multiply()([util_emb, visit_mask[:,:,tf.newaxis]])\n    \n    # date sequence    \n    date_emb = layers.Embedding(input_dim=date_vocab, output_dim=vocab_dim, mask_zero=True, name='date_embedding')(date_seq)\n    date_emb = layers.Multiply()([date_emb, visit_mask[:,:,tf.newaxis]])\n\n    # visit sequence\n    visit_emb = layers.Add()([code_emb, date_emb, util_emb]) \n\n    demo_emb = tf.expand_dims(demo_emb, 1) # (N, 1, emb_size)\n    demo_mask = tf.ones_like(tf.reduce_sum(demo_emb, axis=2), tf.float32) # (N, 1)\n    \n    for trans_layer in range(num_translayer):\n        multihead = multihead_attention(model_dim, num_heads, name=\"multihead_attention-\"+str(trans_layer))(\n            queries=tf.concat([demo_emb, visit_emb], 1),                         \n            keys=tf.concat([demo_emb, visit_emb], 1),\n            values=tf.concat([demo_emb, visit_emb], 1),\n            query_masks=tf.concat([demo_mask, visit_mask], 1),\n            key_masks=tf.concat([demo_mask, visit_mask], 1)\n        )\n        \n        visit_emb = ffn(model_dim, ffn_dim, name=\"ffn-\"+str(trans_layer))(multihead) # (N, max_visit, emb_size)\n            \n        demo_emb = visit_emb[:, :1, :] # (N, 1, emb_size)\n        visit_emb = visit_emb[:, 1:max_visit+1, :] # (N, max_visit, emb_size)\n        \n    patient_embedding = layers.Dense(patient_dim, activation=None, name=\"patient_embedding\")(tf.squeeze(demo_emb, [1]))\n    \n    ##################### NVP task #####################\n    # code_label = layers.Dense(units=model_dim, activation=tf.nn.sigmoid)(patient_embedding)\n    code_label = layers.Dense(units=cat_vocab, activation=tf.nn.sigmoid, name='code_label')(patient_embedding)\n    \n    ##################### CP task #####################\n    # cat_label = layers.Dense(units=model_dim, activation=tf.nn.sigmoid)(visit_emb)\n    cat_label = layers.Dense(units=cat_vocab, activation=tf.nn.sigmoid, name='cat_label')(visit_emb)\n   \n    cls_label = layers.Dense(units=model_dim, activation=tf.nn.relu)(patient_embedding)\n    cls_label = layers.Dense(units=1, activation=tf.nn.sigmoid, name=\"cls_label\")(cls_label)\n        \n \n    outputs = [code_label, cat_label, cls_label]\n    return tf.keras.Model(inputs=inputs, outputs=outputs, name=model_name)\n\nmodel_losses = {\n    \"cat_label\": tf.keras.losses.BinaryCrossentropy(),\n    \"code_label\": tf.keras.losses.BinaryCrossentropy(),\n    \"cls_label\":tf.keras.losses.BinaryCrossentropy(),\n}\n\nmodel_metrics = {\n    \"cat_label\": tf.keras.metrics.Recall(top_k=30),\n    \"code_label\": tf.keras.metrics.Recall(top_k=30),\n}\n\nmodel_weights = {\n    \"cat_label\": 1,\n    \"code_label\": 1,\n    \"cls_label\":0,\n}\n\n\n\n\n\n\n\n\n\n\n\n\n\n    \n"
  },
  {
    "path": "pretraining/plot.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"id\": \"boolean-adaptation\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"import matplotlib.pyplot as plt\\n\",\n    \"import numpy\\n\",\n    \"import _pickle as pickle\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"id\": \"complete-logging\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"def extract_line(lines, substr):\\n\",\n    \"    loss = []\\n\",\n    \"    for line in lines:\\n\",\n    \"        if substr in line:\\n\",\n    \"            l = line.partition(substr)[2].split(\\\" \\\")[0]\\n\",\n    \"            loss.append(float(l))\\n\",\n    \"    return loss\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"id\": \"increasing-composition\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"image/png\": \"iVBORw0KGgoAAAANSUhEUgAAAZQAAAEGCAYAAABCa2PoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA4zElEQVR4nO3de3gV1bn48e+7L0m4hltAIGAAIwiCgBGxVVTUorUVbbFFeyxWW20rbU899UL70yrn0Gr1aK21VioqtXrQYtXUVvEuraVAkIBcJQpIkEsI90uSfXl/f8wKbJKdkIS9s5Ps9/M8+8nMmjVr1uwJeZlZa9YSVcUYY4w5Xr5UV8AYY0zbYAHFGGNMQlhAMcYYkxAWUIwxxiSEBRRjjDEJEUh1BVKpR48empeXl+pqGGNMq7JkyZIdqppTMz2tA0peXh5FRUWproYxxrQqIrIxXro98jLGGJMQFlCMMcYkhAUUY4wxCZHWbSjGmJYlFApRWlpKRUVFqqtigKysLHJzcwkGgw3KbwHFGNNilJaW0qlTJ/Ly8hCRVFcnrakq5eXllJaWMmDAgAbtY4+8jDEtRkVFBd27d7dg0gKICN27d2/U3aIFFGNMi2LBpOVo7LWwgNIEC37/I9780YWE1ryX6qoYY0yLYQGlCba+9i59521mw+y7Ul0VY0ySTJs2jXfeeYeXXnqJX/7yl43at6ysjDPPPJNRo0bxj3/846hteXl57NixI5FVBeDaa69l7ty5tdLnz5/P6NGjCQQCcbcnkgWUJug67acAhKuqUlwTY0yyLFy4kLFjx/Lee+8xbty4Ru371ltvMXz4cJYuXco555yTpBo2TP/+/Xnqqae4+uqrk34sCyhNEMjIAiASjqS4JsaYRLvlllsYMWIEixcv5qyzzuLxxx/ne9/7HtOnT6+Vd8OGDYwfP54RI0ZwwQUX8Omnn1JcXMytt97Kyy+/zMiRIzl06FCt/R5++GFGjx7N8OHDWbNmDQAHDhzguuuuY8yYMYwaNYqXX3758DHOOeccRo8ezejRo/nXv/4FeL2wpk6dyuDBg7nwwgvZvn173PPJy8tjxIgR+HzJ/3Nv3Yab4EhACae4Jsa0XXf/dSWrPtub0DKH9unMz788rN489913H1/72tf44x//yAMPPMB5553H+++/HzfvD37wA6ZMmcKUKVN44okn+OEPf8hLL73E9OnTKSoq4re//W3c/Xr06MEHH3zA7373O+6//34ef/xxZsyYwfjx43niiSfYvXs3Y8aM4cILL6Rnz5688cYbZGVlsW7dOq666iqKiop48cUXWbt2LatWrWLbtm0MHTqU6667DoA777yTgoICLrvssuP7whrJAkoTHIj4yQYObd2LVh1EMtqnukrGmAT64IMPOO2001izZg2nnHJKnfkWLFjAX/7yFwCuueYabr311gaV/5WvfAWA008//fD+r7/+OoWFhdx///2A14X6008/pU+fPkydOpXi4mL8fj8fffQR4LWNXHXVVfj9fvr06cP48eMPlx/vbqo5WEBpgmc+2MUtQPZKP1t+8RX63PVaqqtkTJtzrDuJZCguLubaa6+ltLSUHj16cPDgQVSVkSNHsmDBAtq1a5eQ42RmZgLg9/sJuycdqsoLL7zA4MGDj8p711130atXL5YtW0Y0GiUrKyshdUgGa0NpgpyOJ/O/E/IAOLBzd0rrYoxJnJEjR1JcXMzJJ5/MqlWrGD9+PPPmzaO4uDhuMPnc5z7HnDlzAHjmmWeOqwF+woQJPPzww6gqAEuXLgVgz5499O7dG5/Px9NPP00k4rXdjhs3jueee45IJMKWLVt45513mnzsRLGA0gSj+nflg85nAtaOYkxbU1ZWRteuXfH5fKxZs4ahQ4fWmffhhx/mySefZMSIETz99NM89NBDTT7uHXfcQSgUYsSIEQwbNow77rgDgO9///vMnj378CO4Dh06AHDFFVeQn5/P0KFD+eY3v8lZZ511uKw777yTwsJCABYvXkxubi5//vOfufHGGxk2LHl3flIdDdNRQUGBNmWCrReWlHLXy7N5/vk/ER2XxbCZS5NQO2PSz+rVq+ttszDNL941EZElqlpQM6/doTRBMOAjLBkARCPWddgYY8ACSpNk+IWweMM5B94PUfbYjSmukTHGpJ4FlCY4/cRuREL9eHW0d5eybWH8PurGGJNOLKA0QU6nTGZMPIPH8r8JQDSSvu1QxhhTLakBRUQuFpG1IlIiIrfH2Z4pIs+57QtFJC9m2zSXvlZEJsSkdxGRuSKyRkRWi8hZLv0uEdksIsXu88VknlvQ7zv82CtqQ7AYY0zyXmwUET/wCHARUAosFpFCVV0Vk+16YJeqniQik4F7ga+LyFBgMjAM6AO8KSInq2oEeAh4TVUniUgGEPua+oOqen+yzilW0C8oQcI+iEbtDsUYY5J5hzIGKFHVT1S1CpgDTKyRZyIw2y3PBS4Qb0aXicAcVa1U1fVACTBGRLKBccAsAFWtUtXdSTyHOmX4fYCfsB/Ktwl7X30kFdUwxiRJWxm+/oEHHmDo0KGHB7DcuHFjwo9dLZkBpS+wKWa91KXFzaOqYWAP0L2efQcAZcCTIrJURB4XkQ4x+aaKyHIReUJEusarlIjcICJFIlJUVlbW5JPr1609GupMaQ/I/Uz4+JH/hTR+p8eYtqatDF8/atQoioqKWL58OZMmTWrweGNN0doa5QPAaOBRVR0FHACq22YeBQYBI4EtwP/GK0BVZ6pqgaoW5OTkNLkip/bN5tGrzuM/z/kxn3WDaEQgEmpyecaYlqGtDV9//vnn07691zIwduxYSktLE/I9xZPMwSE3A/1i1nNdWrw8pSISALKB8nr2LQVKVXWhS5+LCyiquq06s4j8AXglYWdSh6ygDyVIyA8aBSJVEMhI9mGNSQ+v3g5bP0xsmScMh0vuqTdLWx6+ftasWVxyySVN+OIaJpkBZTGQLyID8ILBZKDmlGGFwBRgATAJeFtVVUQKgWdF5AG8Rvl8YJGqRkRkk4gMVtW1wAXAKgAR6a2qW1y5VwArknhugNeOouq1o2hUvIBijGn12uLw9X/6058oKirivffea1AdmyJpAUVVwyIyFZgH+IEnVHWliEwHilS1EK9x/WkRKQF24gUdXL7n8YJFGLjJ9fAC+AHwjOvh9QnwLZf+KxEZCSiwAUj66+vBgA80QNgP0ZCPyPaN+PO6JfuwxqSHY9xJJENbHb7+zTffZMaMGbz33nuHj50MSW1DUdW/q+rJqjpIVWe4tDtdMEFVK1T1SlU9SVXHqOonMfvOcPsNVtVXY9KLXRvICFW9XFV3ufRrVHW4S78s5m4laTpkBNBokEMZQuetfj74j69D+cfJPqwxJkna4vD1S5cu5cYbb6SwsJCePXs2uX4N0doa5VuUU3p34uYLRvLYqZNY1V8IHBTYH79hzBjTOrS14etvueUW9u/fz5VXXsnIkSOTOi2wDV/fhOHrY63bto+LHpzPTzb9lLNXVjLyL4/BwHMTVENj0osNX9/y2PD1zSjo975C9Qu+CNZ12BiTtiygHKdgwPsKI34/EgWiFlCMMenJAspxCvoFgEMKgYiwYOp/ESlO+iswxhjT4lhAOU49OmRy1Zj+/KtnASV9hC6blMqFf091tYwxptlZQDlOPp/wy68Mp7TjV3ntQq83iFbZC47GmPRjASVBgn4fYZ/3nqgFFGNMOrKAkiBBv4+Izw+Ahqxh3pjWrq0MX1/thRdeQEQ43lcl6mMBJUEyAj52h70G+lWLV1Lx1lOprZAx5ri0leHrAfbt28dDDz3EmWeemdTjWEBJkLzu7dkQ6k5FEDqvDFF6f+3B2YwxLV9bG74evLfwb7vttiaNA9YYyRxtOK3MmnIGd/21Pd/K7MqvXnmOEyMK0Qi4x2DGmMa5d9G9rNm5JqFlDuk2hNvG3FZvnrY2fP0HH3zApk2buPTSS7nvvvuO7ws8BgsoCeLzCe0y/FRoe0IBiFYPZ+9LzOikxpjm01aGr49Go9x888089dRTjfsCmsgCSgIFfT4iUT9hX/X8KCEIWkAxpimOdSeRDG1t+Pp9+/axYsUKzjvvPAC2bt3KZZddRmFhIQUFtYbiOm7WhpJAQb+PaNRH2A+hsI9Q6WqbZ96YVqStDV+fnZ3Njh072LBhAxs2bGDs2LFJCyZgASWhOmT60WgmFUEhY3uAj748haq/x53a3hjTQrW14eubkw1fn8A+2XsOhXh24XrmvvoYl3/2JhcuruLEmy+g/Q3xG+aMMUez4etbHhu+PkWy2wWZOLIfH8t4Vg/1ZkbTkL01b4xJD0kNKCJysYisFZESEbk9zvZMEXnObV8oInkx26a59LUiMiEmvYuIzBWRNSKyWkTOcundROQNEVnnfnZN5rnVpXp+lKjf9Xewt+aNMWkiaQFFRPzAI8AlwFDgKhGp+TDyemCXqp4EPAjc6/YdCkwGhgEXA79z5QE8BLymqkOA04DVLv124C1VzQfecuvNLqNGQLFxvYwx6SKZdyhjgBJV/URVq4A5wMQaeSYCs93yXOACERGXPkdVK1V1PVACjBGRbGAcMAtAVatUdXecsmYDlyflrI4hGPCGX9l2KArAspc+YPON50DFnlRUxxhjmk0yA0pfYFPMeqlLi5tHVcPAHqB7PfsOAMqAJ0VkqYg8LiIdXJ5eqrrFLW8FesWrlIjcICJFIlJUVlbW5JOrS7ugn6vP7M+OrLNYkh/AVwl7PtgGZWsTfixjjGlJWlujfAAYDTyqqqOAA8R5tKVe17W43ddUdaaqFqhqQU5OTsIrKCL84orhfHXEldw15tssHShoFO+teWOMacOSGVA2A/1i1nNdWtw8IhIAsoHyevYtBUpVdaFLn4sXYAC2iUhvV1ZvoO6R0ppB0O9Do37CfqD6rXljTKvRVoavnz9/PqNHjyYQCNTaPnv2bPLz88nPz2f27Nm19m2sZAaUxUC+iAwQkQy8Rvaab9oUAlPc8iTgbXd3UQhMdr3ABgD5wCJV3QpsEpHqsQkuAFbFKWsK8HIyTqqhggEfqtUBBQsoxrQybWX4+v79+/PUU09x9dVXH5W+c+dO7r77bhYuXMiiRYu4++672bVr13EdK2kBxbWJTAXm4fXEel5VV4rIdBG5zGWbBXQXkRLgZtzjK1VdCTyPFyxeA25S1Yjb5wfAMyKyHBgJ/MKl3wNcJCLrgAvdesoEfAL4ifhAw8KSmfdT9c85qaySMaYB2trw9Xl5eYwYMQKf7+g/9/PmzeOiiy6iW7dudO3alYsuuojXXnvtuL67pA4Oqap/B/5eI+3OmOUK4Mo69p0BzIiTXgzUekNTVcvx7lhahNyu7dFwJ7Zl+/FHlPavbKJ89W30/uuVNqS9MQ2w9Re/oHJ1YoevzzxlCCf89Kf15mlrw9fXZfPmzfTrd6RlITc3l82ba7ZKNE5ra5RvNS4+9QSKfzaJ1YN/zbQfnM/2bIiGscZ5Y1qBxgxfX/0o6ZprruGf//xng8qPHb5+w4YNgDd8/T333MPIkSM577zzDg9fHwqF+M53vsPw4cO58sorWbXKe8p/rOHr6wsmyWLD1ydRdrsgHYLtKQ92pyoAUbUh7Y1pqGPdSSRDWxu+vj59+/bl3XffPbxeWlp6eJj7prI7lCQLBgR1Q9pHrLeXMS1aWxu+/ljHe/3119m1axe7du3i9ddfZ8KECcfesR4WUJIs4PMRiviI+KAy7KNi0yo0VJnqahlj6tDWhq9fvHgxubm5/PnPf+bGG29k2LBhAHTr1o077riDM844gzPOOIM777yTbt26Nbn+YMPXJ3T4+ni+96clvFlayD1vPc9Q9+5/z3Gd6D5zUVKPa0xrZMPXtzyNGb7e2lCS7JYJgxm99uvM2rmDc09Zwhfe2UOobHeqq2WMMQlnj7ySbGBOR75z9lBKMr7EB+dcQEUGaCSa6moZY0zCWUBpJhl+H6ifkN8CijH1SefH8C1NY6+FBZRmEvALql5vr2jY/sEYE09WVhbl5eUWVFoAVaW8vLxR3ZStDaWZBP0+SrZVEPHB1u2ZrJ8wnNPGDaPzT/8PRFJdPWNahNzcXEpLS0nG1BKm8bKyssjNzW1wfgsozeTrBf14d8MI3h/8Dqd/uo9+m8PsfeffdL61wl50NMYJBoMMGDAg1dUwTWQBpZn8ZMJg/ktPZsD6bD6+tIirfjOH7oq9OW+MaTOsDaUZiQhBv6AaIOyDqL05b4xpQyygNLOAzwfqI+KvDig2WKQxpm2wgNLMgn5h+94wYb+w+1CQf97/fQ48dx+EbTgWY0zrZgGlmfXp0o5PtgbZ0Rmy9vro/vwaPr3nD/DJe6mumjHGHBdrlG9mL930eXYfHMOVvx/Ch+ctZ+j8OYxfJRA6kOqqGWPMcbGA0syygn5OyPbTPqMbuzqdxP4swBrnjTFtQFIfeYnIxSKyVkRKROT2ONszReQ5t32hiOTFbJvm0teKyISY9A0i8qGIFItIUUz6XSKy2aUXi8gXk3luxyvgF6JRr3GeKNY4b4xp9ZJ2hyIifuAR4CKgFFgsIoWquiom2/XALlU9SUQmA/cCXxeRocBkYBjQB3hTRE5W1Yjb73xV3RHnsA+q6v3JOqdECvp9VIWFsE8QFfaWbabj7u34OvcAnzVtGWNan2T+5RoDlKjqJ6paBcwBJtbIMxGY7ZbnAheIiLj0OapaqarrgRJXXpvRLuhn2acHqQx665t/+BRrzhrHoQe+ktqKGWNMEyUzoPQFNsWsl7q0uHlUNQzsAbofY18FXheRJSJyQ43yporIchF5QkS6xquUiNwgIkUiUpTK8YJ+dukpTLtwHB+fejWFY/vy2tggokJoc2nK6mSMMcejNT5bOVtVRwOXADeJyDiX/igwCBgJbAH+N97OqjpTVQtUtSAnJ6c56hvXqX2z+c64QYw/dTKPnvBjln2ut1e/UDhldTLGmOORzICyGegXs57r0uLmEZEAkA2U17evqlb/3A68iHsUpqrbVDWiqlHgD7SSR2QBv7sEAa85S8MWUIwxrVMyA8piIF9EBohIBl4je2GNPIXAFLc8CXhbvYkQCoHJrhfYACAfWCQiHUSkE4CIdAC+AKxw671jyr2iOr2ly/B7Q9dr0GtM0XCkvuzGGNNiJa2Xl6qGRWQqMA/wA0+o6koRmQ4UqWohMAt4WkRKgJ14QQeX73lgFRAGblLViIj0Al702u0JAM+q6mvukL8SkZF4bSwbgBuTdW6JFHR3KJv2encmK1cpB744nJO6dmTgPb9B+p2RyuoZY0yDJfXFRlX9O/D3Gml3xixXAFfWse8MYEaNtE+A0+rIf83x1jcVTj+xK2fkdWWHns7i/FJ67aui3Y4wVZ/sJrr0JfwWUIwxrYS9KZ9i+b068efvfg74HF/zXUxEdnHiylv4zrwoGrIBI40xrUdr7OXVZgX9QiTiI+KuilbZ2/PGmNbDAkoLEvT7CEeEsN9b1yq7QzHGtB72yKsFCfp97DwQpa8LKAveX0JgyxRO7XMy2V/7AWR1Tm0FjTGmHhZQWpBenTN5Y1WIvu0zgUP0nH8A5i/iMxbRoVcmgQk/SXUVjTGmThZQWpC7vjyMG84ZxIHQWG6JvsCw7lvwr36Ba96JEt2/J9XVM8aYellAaUECfh/9u7cH2nOg0yls796XrI4vANaeYoxp+axRvoWq7vFlDfTGmNbCAkoLFfT7CMX0+Krcv5fKfeXoof2gmtrKGWNMHBZQWqisoJ9/l+yjyj2U/Ozhf/PJGWezelQBe+/6cmorZ4wxcVgbSgv1s0tPYcHHPVm36yZe3VJIx8p97Mzcx8T5Iao21Ry02RhjUs8CSgs1un9XRvfvCpzENM7m/1ZvJ3fIb5g4f50NcW+MaZHskVcr4L1BH8UfCKLYEPfGmJbJAkor4DXQK0F/BhE/EI6mukrGGFOLPfJqBYJ+HwerwmwsryTkh082wqLLRhBAOLlde/LuuBPfqZekuprGmDTXoDsUN1Oizy2fLCKXiUgwuVUz1T5/UneGnNAZOXgqi4ZkcSAotNsVImt7FZXLdlPxxh9TXUVjjGnwHcp84BwR6Qq8jje979eBbySrYuaIc/JzOOdHOcA5hG64i/yfvcp157VnxeIfcPczETQUSnUVjTGmwW0ooqoHga8Av1PVK4Fhx9xJ5GIRWSsiJSJye5ztmSLynNu+UETyYrZNc+lrRWRCTPoGEflQRIpFpCgmvZuIvCEi69zPrg08t1Yl4PPmoI9GfYSr500J2bwpxpjUa3BAEZGz8O5I/ubS/MfYwQ88AlwCDAWuEpGhNbJdD+xS1ZOAB4F73b5D8eaXHwZcDPzOlVftfFUdqaoFMWm3A2+paj7wlltvc0SEDL+PSNTvNdADWmV3KMaY1GvoI6//BKYBL6rqShEZCLxzjH3GACVuHnhEZA4wEVgVk2cicJdbngv8VkTEpc9R1UpgvYiUuPIW1HO8icB5bnk28C5wWwPOrdUJ+oX1ZYcOD8uybG0plfd9i/b+AEM69qHzxV/B3/+01FbSGJN2GhRQVPU94D0A1zi/Q1V/eIzd+gKbYtZLgTPryqOqYRHZA3R36f+usW/f6uoAr4uIAo+p6kyX3ktVt7jlrUCveJUSkRuAGwD69+9/jFNomXI6ZfLPj3aT2ydIVCJ0Lw5Dsfd1bQWii/9O9z8sTm0ljTFpp0EBRUSeBb4LRPAa5DuLyEOqel8yK1eHs1V1s4j0BN4QkTWqOj82g6qqCzi1uAA0E6CgoKBVjrL48tSzKdtXwe7K0ZSf8RGPPvc2o04K8nbFX3jwDxGih2xkYmNM82toG8pQVd0LXA68CgwArjnGPpuBfjHruS4tbh4RCQDZQHl9+6pq9c/twIt4j8IAtolIb1dWb2B7A8+t1cluF+Sknp0o6Hci48ZexLJO57Kr33g29xCifrE36Y0xKdHQgBJ0751cDhSqagjv0VN9FgP5IjJARDLwGtkLa+QpBKa45UnA26qqLn2y6wU2AMgHFrn3YTqB924M8AVgRZyypgAvN/DcWr2g30c06jWoqF9Qe5PeGJMCDW2UfwzYACwD5ovIicDe+nZwbSJTgXl4PcKecA3604EiVS0EZgFPu0b3nXhBB5fvebwG/DBwk6pGRKQX8KLXbk8AeFZVX3OHvAd4XkSuBzYCX2vgubV63lhf3v8N1C9oxO5QjDHNT7SJkzWJSEBVW/WwtwUFBVpUVHTsjC3c5+95m817d9Np8M/5/cNhuu0/erv4lbxbJ5A15aHUVNAY06aIyJIar20ADW+UzwZ+DoxzSe8B04E9CauhabL/ufxUijft5qODNzH/sn+R+0kJh6oiBDvCvn37uGgphD5eQ1aqK2qMadMa+sjrCby2iurHSNcAT+K9OW9S7PwhPTl/SE/gZJj4Xd5YtY3v/LGI2yYd5NV37+aipTY8izEm+RoaUAap6ldj1u8WkeIk1MckQNDvDc+C+g+//KihVv100hjTCjS0l9chETm7ekVEPg8cSk6VzPHK8HuXVbCAYoxpPg29Q/ku8EfXlgKwiyNddE0LEwx4AeWvy7YdGZ5l6U52ThyBX4SgT8j3tSfntP5kT/sjBDJSWFtjTFvRoDsUVV2mqqcBI4ARqjoKGJ/UmpkmG9ijA6flZrN/Xw6HAv1Y1d+PhoWu20N03l5FcGsllWt2sfO1xVC+LtXVNca0EY2asdG9LV/tZuDXCa2NSYjuHTN5eWr1E8or4EZv6dLf/INunSsolpt5/G85tNu0ESI29L0xJjGOZ055SVgtTLOIfQEy4vdBVCBivb+MMYlxPAGlVQ6smM4yYgJKNOBDo2J3KMaYhKn3kZeI7CN+4BCgXVJqZJImGBC27AlDJmwNHaRLyMcrT95PVeaTtM/w0ykrSH67XnQ+MY+s8d8AX71zqBljzFHqDSiq2qm5KmKSr3uHTN4vKaNj1yw+ztzJsAofg55be1SecmB3RoSTXzoRBp6bmooaY1qlRjXKm9btV5NGcOO5A9ld+Ry7z9zOlrXF+EIVLN64izVb9rE/dz7XLutAXtE2qNyX6uoaY1oZCyhpJCvoZ1ifbLxpZwbCsLEAFM1by7u7P6Z/v6Uc2NwBotvRSJX1ujDGNMrxNMqbNiLo9xGJKkFfkFD1sC1VFamtlDGm1bGAYggGvCAS8AWODNVSaQHFGNM49sjLkBnwosjGHSGW799IAbDy+vtQue+ofD5VAj7o9z+30OHy61NQU2NMS2YBxfClEb3Zc7CKDYeuY+foDyg6uJCge+ExHIlSGY5yoH0VXSN+ChYeonJVMR0uT22djTEtjwUUQ6/OWdz8hcHAYOAquO7ItmcWbuS/X1zBeee+iG//HgoWrkBD9jKkMaa2pLahiMjFIrJWREpE5PY42zNF5Dm3faGI5MVsm+bS14rIhBr7+UVkqYi8EpP2lIisF5Fi9xmZzHNLF0E3FL6PAJU+b656rbKAYoypLWl3KCLiBx4BLgJKgcUiUqiqq2KyXQ/sUtWTRGQycC/wdREZCkwGhgF9gDdF5GRVjbj9fgSsBjrXOOwtqjo3WeeUjqrnVvFLgEpfFAAN29wqxpjakvnIawxQoqqfAIjIHGAiEBtQJgJ3ueW5wG9FRFz6HFWtBNaLSIkrb4GI5AKXAjPwRjw2SVR9h/LRtkPs5TOioqz95xLmf2XkUfky/EJ7n58hvg5kDsynx4zHU1BbY0wqJfORV19gU8x6qUuLm0dVw8AeoPsx9v01cCsQjXPMGSKyXEQeFJHMeJUSkRtEpEhEisrKyhp3RmloSO9OnNSzI4f2DCASbsfyQT4IKb1KK4/6ZH9aQWDDAfat3EbZC+8TraxMddWNMc2sVTXKi8iXgO2qukREzquxeRqwFcgAZgK3AdNrlqGqM912CgoKbMTkYxiU05E3bz4XOBf4f95Dyhi7D1YxcvobXDFuE2+WPcJfPxzAoVfWoaEQZMaN6caYNiqZdyibgX4x67kuLW4eEQngjQlSXs++nwcuE5ENwBxgvIj8CUBVt6inEngS7xGZSbLqR2Kq3rss0aD3U0M2z4ox6SaZAWUxkC8iA0QkA6+RvbBGnkKOzE0/CXhbVdWlT3a9wAYA+cAiVZ2mqrmqmufKe1tV/wNARHq7nwJcDqxI4rkZpzqgoN7NbiQQ9FYtoBiTdpL2yEtVwyIyFZgH+IEnVHWliEwHilS1EJgFPO0a3XfiBQlcvufxGvDDwE0xPbzq8oyI5ODN1VIMfDcZ52WOFnRjf23b4wWQ5VV7GQq8/ug0wh2zjsorIvTomEHPzGxOyOiCr2d/2l9wRXNX2RiTJOLdEKSngoICLSoqSnU1Wr1R019nL2tpf+JMRq+LcvvceP0l4hv4t1fIHDQoibUzxiSaiCxR1YKa6a2qUd60TK/+aByf7S6g9MBZVI08wKfDFiKVR7/8GI7CS0s306X/VjZG1vO7nQPZXria6O6dgAUUY9oCCyjmuJ2QncUJ2VmMppuXcFrtmR4PVUX4/kevcd5JS1i7bz1Z3fOB1WjlweatrDEmaWz4etMsqttawE9EI2hGBmDDuBjTllhAMc3C7xNEgMPdi72bY608lMJaGWMSyR55mWYhImT4ffz74z0Ee8I3P32BXwIf/9dtRG65rd59fUA7VcQv9Jv1R9qdfkaz1NkY0zgWUEyz+e/LT2Xp5o6sq/AR7LePxZ9fRGZF3b3BQxFlny/Efn+Iq6s6s3fZXipXLbOAYkwLZQHFNJuvFfTjawX9gHO8hG/Un3/aXz7k1U3PEOnyN76fO4W9Nz2M2lz3xrRY1oZiWqwMvxCNer+ikQx7A9+Yls4Cimmxgn4fkYj3KxquDih2h2JMi2WPvEyLFQz4CEV8ZAI3ffgSdwIrXniGsn8838AShEGZHTjBlwkdepD97Vtof/rpSayxMenNAoppsUb160KXFXlUhLuxWrawth/02hmh7/5jDevmUYGwHmJ/NEq4opRou2csoBiTRBZQTIv1hWEn8IVh1wHXeQnX15v9KMtLdzP5r9/glBNyeKFdHz65a661vxiTZNaGYtqkoN+Hqp9INAz+IOJTNBROdbWMadMsoJg2Kej3gfoJaQj8GYhE7Q7FmCSzR16mTcrw+wA/uyp28dfyzeQFhNCG1Sz/7U+aVJ4fHyO69aNLVga+fsPJKjgLby43Y0w1CyimTeqUFYBIe/ZF1vLTg1v4SXYGYz7ayaDf/q3JZe51H4ATn/mTNfAbU4MFFNMmde2QwZyv3svy7ashGqZD95WsLy1rUlkVWsmcHfPof3AEPysvZut8JbJn77F3NCbNWEAxbdZpfXtzWt/ebu0LTS5nT+Uefj7ndTbuO5l27UqAfWjY2mOMqSmpjfIicrGIrBWREhG5Pc72TBF5zm1fKCJ5MdumufS1IjKhxn5+EVkqIq/EpA1wZZS4MjOSeW4mfQR93lv6EQ0jQRsCxpi6JC2giIgfeAS4BBgKXCUiQ2tkux7YpaonAQ8C97p9hwKTgWHAxcDvXHnVfgSsrlHWvcCDrqxdNOqtBWPqdnRAcRODWUAxppZkPvIaA5So6icAIjIHmAisiskzEbjLLc8Ffite15mJwBxVrQTWi0iJK2+BiOQClwIzgJtd2QKMB652Zc125T6arJMz6SPg8/6ZhDq9zsS9IR4A1t8xjdDd0xJ2jExVMtVbFvHRYdyF5P7mNwkr35jmkMyA0hfYFLNeCpxZVx5VDYvIHqC7S/93jX37uuVfA7cCnWK2dwd2q2o4Tv6jiMgNwA0A/fv3b9QJmfQkInxr8I95d/0KsmUrCz+/jg77owkrf7uvkk4aYHg4Gzm0m847DnBw0aKElW9Mc2lVjfIi8iVgu6ouEZHzmlKGqs4EZgIUFBRo4mpn2rKbx17HzWPdyrWJLXvKq1MI+oJcce6jTP/vn/JfH85hz2f2Vr9pfZLZKL8Z6BeznuvS4uYRkQCQDZTXs+/ngctEZAMwBxgvIn9y+3RxZdR1LGNapKAvSCgaIuj3EdIA4sOGiTGtUjIDymIg3/W+ysBrZC+skacQmOKWJwFvq6q69MmuF9gAIB9YpKrTVDVXVfNceW+r6n+4fd5xZeDKfDmJ52ZMwgT8AcLRsBdQCHjjjlm3ZNMKJe2Rl2sTmQrMA/zAE6q6UkSmA0WqWgjMAp52je478YIELt/zeA34YeAmVT3WmOW3AXNE5H+Apa5sY1q8oATZtG8T9y7+Jet6fsg/N2VxSiTKnyefjpL84V2CCHmRDvjcsVSEbe1PYt0Fk7nu6+Non9GqnoybFBLvP/fpqaCgQIuKilJdDZPmZn04iydXPgnAwYoqxnx0gG+9FsWXuHb/einQMar48P4W+IBIlY9nB1/IhPt/ztn5PZqnIqbVEJElqlpQM93+62FMil0//HquH17jtalfNM+x39v0HlPfnsqcS+cwpMcwL/GX/Vj9TGcC0QihSDNFNdMm2PD1xqSx6pc2Q9GYNht/EPELgWiEKgsophEsoBiTxoL+eAElwwWUsN2hmEaxR17GpLHqUQBWl68+vExGkAxRBlV9zNZ5DzJvUfvD+dv729Ej0AWAqD+TUJcBDMrpiN/v/d/U36ULmQMHNus5mJbDAooxaaxzRmcA7iu6LyYRHsiKMGzzNpj1Sh17eo83MvGGpYiV/4/5BHJyEl5X0/JZQDEmjQ3MHsizX3yWfaF9RxIP7MDXaxWffrydqvCRR14fV25m0cFVTM2ZRI/KffT56E88EPoqgwsu4NIRvTm4YAHlj88ieuAAWEBJSxZQjEljIsLwnOG1N+RfVivphY9eYNaCuxg46XucsGcLzJzFxqre5Jwyio6fH0SkvByAdH4VId1Zo7wxpkG8Qb2PDhgCKIeHSfZ+WjxJWxZQjDENIu5N+ihRiHmD/0h8qU6ziJKuLKAYYxrl6DuUmOBx+A7FAkq6soBijGmQw4+8atyBHA4wh29QLKCkKwsoxpgG8Yn7c6EcvhsR9HD8ELtDSXsWUIwxDVLdhlLrDuVwhtqN9ia9WEAxxjRKVOtolLdeXmnPAooxpkHitaEc1W3YenmlPQsoxpgG8bk/F7Ub5d2CtaGkPQsoxpgGkdhHWnLkbqT2aygWUNJVUgOKiFwsImtFpEREbo+zPVNEnnPbF4pIXsy2aS59rYhMcGlZIrJIRJaJyEoRuTsm/1Misl5Eit1nZDLPzZh0U1ejfEw3r6PXTdpJ2lheIuIHHgEuwhuQdLGIFKrqqphs1wO7VPUkEZkM3At8XUSG4s0vPwzoA7wpIicDlcB4Vd0vIkHgnyLyqqr+25V3i6rOTdY5GZPWDscLPbwicqTFJN7QLCa9JPMOZQxQoqqfqGoVMAeYWCPPRGC2W54LXCDeb+VEYI6qVqrqeqAEGKOe/S5/0H3st9eYZnD00Cueo/6AiGDSWzIDSl9gU8x6qUuLm0dVw8AeoHt9+4qIX0SKge3AG6q6MCbfDBFZLiIPikhmAs/FmLRX/WKjqh4VPKzbsKnW6hrlVTWiqiOBXGCMiJzqNk0DhgBnAN2A2+LtLyI3iEiRiBSVlZU1R5WNaROE2ncgPuJ0G7ZHXmkrmQFlM9AvZj3XpcXNIyIBIBsob8i+qrobeAe42K1vcY/EKoEn8R651aKqM1W1QFULcmwSIGMaLF6jvEjsHUp1qgWUdJXMgLIYyBeRASKSgdfIXlgjTyEwxS1PAt5Wr0WvEJjseoENAPKBRSKSIyJdAESkHV6D/xq33tv9FOByYEUSz82Y9BOnUR5qD71idyjpK2m9vFQ1LCJTgXmAH3hCVVeKyHSgSFULgVnA0yJSAuzECzq4fM8Dq4AwcJOqRlzQmO16kPmA51W1etLrZ0QkB+83vRj4brLOzZh0FG8+FAEbHNIcltQpgFX178Dfa6TdGbNcAVxZx74zgBk10pYDo+rIP/5462uMqVu80YbD0Si/f+9jbjp/kAUU0/oa5Y0xqRG3DcX9XLd9v402bCygGGMa5tgvLlq34XRnAcUY0yjeHcrRXYi1xvheJj1ZQDHGNEh1G4o3H4qnek75SFRtcEiT3EZ5Y0zbUd2Gcs2r1/DtQVcgXbM5mT9yM3/k34VQvKU75wAbv/EfzLy+B8Hu2bXK6Latki7lVXwytFOtbZ13VvHVxzex4oxsIv7WMYxL708PsbNnJvnL97K7Rwa7e2SwpX87znxrB8/+cADRJpxHxz0hLn9iE+W9MjmhtAKAZWO7HN5+2r93A/CX6/uxp3sGHfaGOHnZPiROIC8d2J6zXy0je1eIshMyyaiMMv9LPdnRO4srCqZy1oiLm3TedbGAYoxpkJ0VOw8vP/7xiwSyOx+1/dI1lYeXb5i1g6tv31WrjEf/HCH7IDx2anmtbc8+HgHg1MV7CLeCZyc+9T49P/POu/v2Krpvr2LQKm+4wU4L1jN3XONP5O6/RgiGOBxMwAsiYR8Ejtwc8pVZm7j6dj+Xr4wycoHW+s4CUei4YTfZ7jLkbPXqecHzm/n2j/2M2LzMAooxJjUqI0cCxoU9z+DBhS8ctX1rWUd2cSTILL229rvFq+85BYAPpnx4pJtxjW2ZgwdzyssvJaraSXOouJgNk6+qc/t1fSfxs2unN7rc6u8hVt8HH6DzJZewc/Zstv3ynsPpS69dwfbdv6b8X48zfOXR3/fGa7/FkC2fEeLTo9LbV8a/NonQCv4fYIxpCcLR8OHloNT+09GoPybhcJ2bxO9vTEmpEwzWu1mjkcQdq/r7jnNMDYWQOOkSDBI9eDBxdWgACyjGmAaJ6JE/kBm+OH/AGlGW1hNQCLSOByfx/ogfpb5zbPTBvG/Xl5FRa1N9AUUPNG9AaR1XzhiTcrF3KBlS+y6i5nQoFWs/qrOsitVr8HXoEH9j62iPP2ZAqdq8ud7voFF8EveYFWs/IrJjBxInCEswSPTQobjFVaz9iGDfvvg71nENmsgCijGmQfp36n94eUjH3FrbMzof/T/y9RNrzqd3xMarr65zW4cxZzahds3P37FjvdsPFS2p9ztojOAJvQHIOPHEo9Kryw/261drH1+njnV24V4/cSL9/jCTjueck5D6VbOAYoxpkHP7ncvdn7ub8kPlTOo05HD6oTN/SLuqXXS+7ho+m/sGO5YtpseFE+jT86RaZWhVJdH9+/F36177AJEwB5cWk3Pzj5N5GgkTyMnhhP+eTmjjRoL9+9PhzDMJ79hBxZo17HttHl2/8Y0mzWKpoRCVJevoeuWV7Jj5B3wdO9BuuDftU7uRI+n105+yf/58Ol9yCT4X1DIHDaxVTs8f/5iO54zjwIJ/Ed5eRpdJX6Vi1Woy8vKQYJCsIUNq7XO8JJ3H3SkoKNCioqJUV8OY1mf9fJj9ZTjxbPjW31JdG9PMRGSJqhbUTLdGeWNM41W/Le+zPyHmCPttMMY0XnWX2Djdh036st8GY0zjHX4vIrG9hEzrZo3yxpjGG3AunH0zjP1+qmtiWhALKMaYxvP54MKfp7oWpoVJ6iMvEblYRNaKSImI3B5ne6aIPOe2LxSRvJht01z6WhGZ4NKyRGSRiCwTkZUicndM/gGujBJXZu1XSo0xxiRN0gKKiPiBR4BLgKHAVSIytEa264FdqnoS8CBwr9t3KDAZGAZcDPzOlVcJjFfV04CRwMUiMtaVdS/woCtrlyvbGGNMM0nmHcoYoERVP1HVKmAOUPO10YnAbLc8F7hAvCFIJwJzVLVSVdcDJcAY9ex3+YPuo26f8a4MXJmXJ+m8jDHGxJHMgNIX2BSzXurS4uZR1TCwB+he374i4heRYmA78IaqLnT77HZl1HUs3P43iEiRiBSVlZU1/eyMMcYcpdV1G1bViKqOBHKBMSJyaiP3n6mqBapakJOTk5Q6GmNMOkpmQNkMxI5YluvS4uYRkQCQDZQ3ZF9V3Q28g9fGUg50cWXUdSxjjDFJlMyAshjId72vMvAa2Qtr5CkEprjlScDb6g0uVghMdr3ABgD5wCIRyRGRLgAi0g64CFjj9nnHlYEr8+XknZoxxpiakvYeiqqGRWQqMA/wA0+o6koRmQ4UqWohMAt4WkRKgJ14QQeX73lgFRAGblLViIj0Bma7Hl8+4HlVfcUd8jZgjoj8D7DUlW2MMaaZpPVowyJSBmxMdT2aoAewI9WVaEbpdr5g55wuWus5n6iqtRqh0zqgtFYiUhRv6Oi2Kt3OF+yc00VbO+dW18vLGGNMy2QBxRhjTEJYQGmdZqa6As0s3c4X7JzTRZs6Z2tDMcYYkxB2h2KMMSYhLKAYY4xJCAsoLYCI9BORd0RklZvn5UcuvZuIvCEi69zPri5dROQ3bu6X5SIyOqasKS7/OhGZUtcxWwI30OdSEXnFrced06ax8+a0ZCLSRUTmisgaEVktIme15essIj92v9MrROT/3JxGbe46i8gTIrJdRFbEpCXsuorI6SLyodvnNyIizXuGDaSq9knxB+gNjHbLnYCP8OaQ+RVwu0u/HbjXLX8ReBUQYCyw0KV3Az5xP7u65a6pPr96zvtm4FngFbf+PDDZLf8e+J5b/j7we7c8GXjOLQ8FlgGZwADgY8Cf6vM6xjnPBr7tljOALm31OuON+L0eaBdzfa9ti9cZGAeMBlbEpCXsugKLXF5x+16S6nOO+z2kugL2iXNRvHHILgLWAr1dWm9grVt+DLgqJv9at/0q4LGY9KPytaQP3gCeb+HNY/OK+4eyAwi47WcB89zyPOAstxxw+QSYBkyLKfNwvpb4wRv8dD2uM0zN69fWrjNHpqHo5q7bK8CEtnqdgbwaASUh19VtWxOTflS+lvSxR14tjLvNHwUsBHqp6ha3aSvQyy3XNV9MQ+agaSl+DdwKRN16fXPaNHrenBZqAFAGPOke9T0uIh1oo9dZVTcD9wOfAlvwrtsS2v51rpao69rXLddMb3EsoLQgItIReAH4T1XdG7tNvf+atIk+3iLyJWC7qi5JdV2aWQDvscijqjoKOID3KOSwNnadu+LNvjoA6AN0wJtuIu20petaHwsoLYSIBPGCyTOq+heXvM2NsIz7ud2l1zVfTEPmoGkJPg9cJiIb8KaGHg88RN1z2jR53pwWphQoVW+WUfCmrB5N273OFwLrVbVMVUPAX/CufVu/ztUSdV03u+Wa6S2OBZQWwPXYmAWsVtUHYjbFzhcTO8dLIfBN11tkLLDH3VrPA74gIl3d/w6/4NJaFFWdpqq5qpqH1/j6tqp+g7rntGnUvDnNdBqNpqpbgU0iMtglXYA3RUObvM54j7rGikh79ztefb5t+jrHSMh1ddv2ishY9z1+k5Y631OqG3HsowBn490OLweK3eeLeM+P3wLWAW8C3Vx+AR7B6+3yIVAQU9Z1QIn7fCvV59aAcz+PI728BuL9oSgB/gxkuvQst17itg+M2f9n7ntYSwvt+VLjfEcCRe5av4TXm6fNXmfgbmANsAJ4Gq+nVpu7zsD/4bUThfDuRK9P5HUFCtx3+DHwW2p07GgpHxt6xRhjTELYIy9jjDEJYQHFGGNMQlhAMcYYkxAWUIwxxiSEBRRjjDEJYQHFmBpE5Jcicr6IXC4i0xq5b44bKXepiJxTY9vjIjLULf80wXW+VkT6xDuWMc3Fug0bU4OIvA1cCvwCmKuq7zdi38nAhar67WPk26+qHRtZL7+qRurY9i7wE1UtakyZxiSS3aEY44jIfSKyHDgDWAB8G3hURO6MkzdPRN5281m8JSL9RWQk3pDlE0WkWETa1djnXREpEJF7gHYuzzNu23+IyCKX9piI+F36fhH5XxFZBpwlIneKyGLx5heZ6d62noT34tsz1cetPpYr4yo3l8YKEbk3pj77RWSGiCwTkX+LSC+XfqXLu0xE5if8izZtV6rfrLSPfVrSBy+YPAwEgffryfdXYIpbvg54yS1fC/y2jn3exb0VDeyPST/FlRd0678DvumWFfhaTN5uMctPA1+uWXbsOt6gjJ8COXiDU74NXB5TdvX+vwL+n1v+EOjrlruk+prYp/V87A7FmKONxpvMaQiwup58Z+FNDgbeH/azj+OYFwCnA4tFpNitD3TbIniDhlY737XRfIg3qOawY5R9BvCuegM0hoFn8CaDAqjCm6MEvGHl89zy+8BTIvIdwN/UkzLpJ3DsLMa0fe5x1VN4I7nuANp7yVKMN5nToWQeHpitqvE6AFSoazcRkSy8u5cCVd0kInfhjX/VVCFVrW5EjeD+Hqjqd0XkTLx2pCUicrqqlh/HcUyasDsUYwBVLVbVkRyZfvltYIKqjqwjmPwLb6RkgG8A/2jkIUPiTVkA3gCCk0SkJxyei/zEOPtUB48d4s2dMylm2z686aNrWgScKyI9XLvMVcB79VVMRAap6kJVvRNvQrB+9eU3pprdoRjjiEgOsEtVoyIyRFVX1ZP9B3gzL96C90f3W4083ExguYh8oKrfEJH/B7wuIj68EWtvAjbG7qCqu0XkD3ijzm4FFsdsfgr4vYgcwnscV73PFhG5HW/IeAH+pqrHGvr8PhHJd/nfwnsEaMwxWbdhY4wxCWGPvIwxxiSEBRRjjDEJYQHFGGNMQlhAMcYYkxAWUIwxxiSEBRRjjDEJYQHFGGNMQvx/JY1DnV0v3qcAAAAASUVORK5CYII=\\n\",\n      \"text/plain\": [\n       \"<Figure size 432x288 with 1 Axes>\"\n      ]\n     },\n     \"metadata\": {\n      \"needs_background\": \"light\"\n     },\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"history_list = []\\n\",\n    \"for i in [1,2,4,10]:\\n\",\n    \"    \\n\",\n    \"    with open(\\\"diff_head/\\\" + str(i) + '.out') as f:\\n\",\n    \"        lines = f.readlines()\\n\",\n    \"        loss = extract_line(lines, \\\"loss: \\\")\\n\",\n    \"        x1, x2 = 1000,11000\\n\",\n    \"        plt.plot(range(x1, x2), loss[x1:x2], label=\\\"# of head:\\\"+str(i))\\n\",\n    \"plt.legend()\\n\",\n    \"plt.xlabel(\\\"# of iterations\\\")\\n\",\n    \"plt.ylabel(\\\"Loss\\\")\\n\",\n    \"plt.savefig(\\\"diff_head\\\", dpi=600)\\n\",\n    \"plt.show()\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"id\": \"under-unemployment\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"image/png\": \"iVBORw0KGgoAAAANSUhEUgAAAY4AAAEGCAYAAABy53LJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABE2UlEQVR4nO3deXhU1fnA8e87S2ayQBLCTljCLsiOLCqKUgFxQfyhoqi41bpXa61iLS6VFtTautStaFVEUWm1WBdQQECKSED2TXbCGgJkIXvm/P64lxhCEjLJTCbJvJ/nycPMueeee24G5uXec+57xBiDUkopVVmOUHdAKaVU3aKBQymllF80cCillPKLBg6llFJ+0cChlFLKL65Qd6AmNG7c2LRr1y7U3VBKqTpjxYoVh40xTcraFhaBo127diQnJ4e6G0opVWeIyK7ytumtKqWUUn7RwKGUUsovGjiUUkr5JSzGOJRSoVFQUEBKSgq5ubmh7ooqh9frJTExEbfbXel9NHAopYImJSWFBg0a0K5dO0Qk1N1RpRhjSEtLIyUlhaSkpErvp7eqlFJBk5ubS0JCggaNWkpESEhI8PuKUAOHUiqoNGjUblX5fDRwVGD55ztY9c1ufD5NPa+UUido4KjAD5/tYMmsrRzZlxXqriilqmnixIksWLCATz/9lD//+c9+7ZuamsrAgQPp06cPixcvPmnb0KFDa8UDxiNHjiQuLo5LL7006MfSwFGBy+7tBUBBblGIe6KUqq5ly5YxaNAgFi5cyHnnnefXvvPmzaNHjx78+OOPDBkyJEg9LFtRUeW+fx566CGmT58e5N5YNHBUwBVh/XpysgpC3BOlVFU99NBD9OzZk+XLlzN48GCmTZvGnXfeyVNPPXVK3Z07d3LhhRfSs2dPhg0bxu7du1m1ahW/+93v+M9//kPv3r3Jyckp91h33nkn/fv3p3v37jz++OMAzJ8/nyuuuKK4ztdff82YMWMAmDt3LoMHD6Zv375cddVVZGVZdzfatWvHww8/TN++ffn444958cUX6datGz179mTcuHFlHnvYsGE0aNCgqr8mv+h03ApERFq/ni9fW8sv/3YeEV79dSlVVU9+tp4N+zIC2ma3lg15/LLuFdZ59tlnufrqq3n33Xd5/vnnGTp0KEuWLCmz7r333suECROYMGECb731Fvfddx+ffvopTz31FMnJybz88ssVHmvy5Mk0atSIoqIihg0bxpo1a7jgggu46667SE1NpUmTJvzzn//klltu4fDhwzz99NN88803REdHM3XqVJ5//nkmTZoEQEJCAitXrgSgZcuW7NixA4/Hw7FjxwBITk7mtddeY9q0aX7+1qpPrzgqkNAqhjbdGwGQq1cdStVZK1eupFevXmzatIkzzjij3HpLly7luuuuA+CGG27gu+++8+s4H330EX379qVPnz6sX7+eDRs2ICLccMMNvPfeexw7doylS5dy8cUX8/3337NhwwbOOeccevfuzTvvvMOuXT/nFbzmmmuKX/fs2ZPx48fz3nvv4XJZ/4Ht379/SIIG6BVHhUSEroNbsHv9EQoLfKHujlJ12umuDIJh1apV3HTTTaSkpNC4cWOys7MxxtC7d2+WLl1KZGRkwI61Y8cOnnvuOZYvX058fDw33XRT8fMRN998M5dddhler5errroKl8uFMYaLLrqIDz74oMz2oqOji19//vnnLFq0iM8++4zJkyezdu3a4gASCnrFUYGU+x+gYON6AL6dsQljdFquUnVJ7969WbVqFZ07d2bDhg1ceOGFzJkzh1WrVpUZNM4++2xmzpwJwIwZM/waCM/IyCA6OprY2FgOHjzIl19+WbytZcuWtGzZkqeffpqbb74ZgEGDBrFkyRK2bt0KwPHjx9myZcsp7fp8Pvbs2cMFF1zA1KlTSU9PLx4LCRW94qhA5ldf4S50AyPZvzWdnMwCohpGhLpbSik/pKamEh8fj8PhYNOmTXTr1q3cui+99BI333wzzz77bPF4RGX16tWLPn360LVrV1q3bs0555xz0vbx48eTmppafKusSZMmvP3221x77bXk5eUB8PTTT9O5c+eT9isqKuL6668nPT0dYwz33XcfcXFxp4xxDBkyhE2bNpGVlUViYiJvvvkmI0aMqHT//SHh8L/o/v37m6rMs95x1dU44+LIuukJFkzfxA2TB9MwIXCXtkrVdxs3bqxwTCGc3HPPPfTp04dbb7011F05RVmfk4isMMb0L6u+3qqqgMPrJfv773H6rIHxA9vTQ9wjpVRd1K9fP9asWcP1118f6q4EhN6qqoCzcQKmoIDCxV8D7fn6zQ20PqMRkTF6u0opVXkrVqwIdRcCSq84KtDij38EoFH+Xnpf1AaAvOOFoeySUkqFnAaOCjhjYnA1bw55eTRr1xBAp+UqpcKeBo7TcHg8pH/yCYVbNgAwa0oyOVn5Ie6VUkqFjgaO02h8150ANNi3hrhmURQV+kg/VH6uGqWUqu80cJxG7OjRSFQUroIcho7vAkBhvmbLVaquqc9p1VetWsXgwYPp3r07PXv25MMPPwzq8TRwVILD7ebI9OmQeQyAlXN2kZ+rg+RK1SX1Oa16VFQU7777LuvXr+err77i/vvvL06GGAwaOCrB06kTFBVRNO+/AOzZeJSdaw6HuFdKqcoIh7TqnTt3plOnToCV3qRp06akpqZW+Xd2OkF9jkNERgIvAE5gmjFmSqntHuBdoB+QBlxjjNlpb5sI3AoUAfcZY+bY5Q8AtwEGWAvcbIzxb6V1P7V59x029+1HRH4GN/7pbN599H8U5OntKqX88uUjcGBtYNts3gMunlJhlXBLq/7DDz+Qn59Phw4dKvMbrJKgXXGIiBP4O3Ax0A24VkRKJ4m5FThqjOkI/BWYau/bDRgHdAdGAq+IiFNEWgH3Af2NMWdiBaSyVzUJ5Lk4HDgaxFCYdrh4cae8bL1VpVRdES5p1ffv388NN9zAP//5TxyO4N1QCuYVxwBgqzFmO4CIzARGAxtK1BkNPGG/ngW8LCJil880xuQBO0Rkq93ebrvPkSJSAEQB+4J4DsWc0TFkfvkVrrYdEOnG0k+2UVTo46xLkmri8ErVfae5MgiGcEqrnpGRwSWXXMLkyZMZNGhQwM6rLMEc42gF7CnxPsUuK7OOMaYQSAcSytvXGLMXeA4rgOwH0o0xc8s6uIjcLiLJIpIciHt9LSY/DUDR7h2M/FUPvNFunZarVC0XLmnV8/PzGTNmDDfeeCNjx46tdJ+rqk4NjotIPNbVSBLQEogWkTKzhhlj3jDG9DfG9G/SpEm1jx3Vrx+eLl3w5eXRvncTIhu4KSzQcQ6lajt/06r/85//pGfPnkyfPp0XXnih0scpmVb9uuuuKzOteuvWrctMq96zZ08GDx7Mpk2bTmn3RFr1Hj160KdPn5PSqt92222AdYts0aJFvP322/Tu3bs4YAZL0NKqi8hg4AljzAj7/UQAY8yfS9SZY9dZKiIu4ADQBHikZN0T9YBEYKQx5la7/EZgkDHmror6UtW06qXtuOYaCvftp+lvH2TO+pZkp+fRa1gb+gxvU+22laqPNK36zzSteuUsBzqJSJKIRGANYs8uVWc2MMF+PRaYb6xINhsYJyIeEUkCOgE/YN2iGiQiUfZYyDBgYxDP4SQNhg6l6Phx0qa9SbseCRQW+Pjhs+01dXilVB2ladUryRhTKCL3AHOwZj+9ZYxZLyJPAcnGmNnAm8B0e/D7CPYMKbveR1gD6YXA3caYImCZiMwCVtrlPwJvBOscSmt8553kbdtOzpo1DLisPcZA8hc7McZgxTGllDpVfUurHtTnOIwxXwBflCqbVOJ1LnBVOftOBiaXUf448Hhge1p54vXgy8wkf9eu4qm5+7el07x9LA6HBg+lVP1XpwbHawNXfDxFR4+yffQVeL3Wr++T51ay/cfgPaWplFK1iQYOPzW+807ix4/H5ObSqVdDRt3ZA4BcTbWulAoTGjj85IiKwtOpIwBSkE/LzvGALvCklAofGjiqQDxeAPbcehuFO7YBVsbche9vDmW3lFIVqM9p1U/IyMggMTGRe+65J6jH0cBRBdGDB9Fg+HDyfvqJgg1r6T+qHW6vi60rD4W6a0qpctTntOon/OEPf/D73KpCA0cVuJs3p/kT1sQuX24eAy9vT1Kvxnq7SqlaKBzSqoM15ffgwYMMHz68qr+qSgvqdNz6zOHxAHB86VIie/bA5Y6mML+Itd+m0L5PE6JjPSHuoVK1y9QfprLpyKkpNaqja6OuPDzg4QrrhENadZ/Px4MPPsh7773HN9984+dv0X96xVFFEhmJq2lTsubNY/8fJhHbJAoMLJq5hTXz95y+AaVUjanvadVfeeUVRo0aRWJiol/9rSq94qgicTjo8M3X7Hv4YXLXrOWMs1vQvndjpv9hKfm5mvxQqdJOd2UQDOGSVn3p0qUsXryYV155haysLPLz84mJiWHKlOCkstcrjmpwRETgbNAQX14epqgIT5Qbd4STwrwifL7gJI9USlVeuKRVnzFjBrt372bnzp0899xz3HjjjUELGqCBo9ocUVEUHT7M5t59yFm7DrfHyabvD/DGrxdyZN/xUHdPqbAXDmnVa1rQ0qrXJoFKq16W/JQUjk5/jyPvvEPLvzxHZofBbFuZyrpFexl1V0+SejYOynGVqgs0rfrPNK26KhaRmEijG28AwOTmkti1ET2GWgNUhfk61qGU0rTqqgzitZ4kPzbrXxSlZ+C+3JpnvW7hXg6nZDHo8vaIZs5VKmzVt7TqesURAM7YWCL79CFvyxYOPfMMkTEumrZtQNreLFZ+tYv0VF2bXClVf2jgCABxuWj3wfsk3PErAJy+Aq6aeBYXXN8V0ASISqn6RQNHBTambSSnsPJXCw47+WHO6tUUZWbidFu/3sN7Mjl2KDsofVRKqZqmgaMCV//3ap5c+mSl6zvj4wDYffMtpNx3H94YNwDz3tnIjEnfc+ygBg+lVN2ngeM00nLSKl234YgRtP7HP4js3Zuiw2k0a9eQKx7ow8DLkwDIztDFnpQKlfqeVn3kyJHExcVx6aWXnlS+Y8cOBg4cSMeOHbnmmmvIz6/+95AGjgoMbD6QvKK8SteXiAhihpyLOzERX14eIkKrLvG0Kl7sSafnKhUq9T2t+kMPPcT06dNPKX/44Yd54IEH2Lp1K/Hx8bz55pvV7pMGjgp4XB5+PPQjc3fO9Ws/8Xoo2L+fHWOv4ujMD3FFOAFY9MEWPp6SzK71lb+KUUpVT7ikVR82bBgNGjQ4qcwYw/z58xk7diwAEyZM4NNPP6307648+hxHBa7oeAWLUhaxMGUhw9tVPsd97CWXUJR6mOwffyRz7lxaXjmWzgOakZddyJ4NR9i9Lo223ROC2HOlap8Df/oTeRsDm1bdc0ZXmj/6aIV1wiGtennS0tKIi4srToiYmJjI3r17KzyHytArjgpc1PYikmKTyC3M9Wu/6MGDaf36a3jPOANfXh6uCCcX3dKdS+/phTfGrdNzlaph9T2tek3TK47T8Dq9zN01l7ScNBIi/btKEK+Hgq0pHPvXv4js1QtPx464Ihwc2ZfFhiX7aNDIS+szGgWp50rVLqe7MgiGcEmrXp6EhASOHTtGYWEhLpeLlJQUWrVqVe1z1SuO02gR3QKAdza84/e+7pYtKdi7l/2/f4z9TzwBQINGXg5sz2DB9E189uIqCjSflVJBEy5p1csjIlxwwQXMmjULgHfeeYfRo0dX+pzKE9TAISIjRWSziGwVkUfK2O4RkQ/t7ctEpF2JbRPt8s0iMsIu6yIiq0r8ZIjI/cE8h+eGPgdAVn7lPqiSmj/2GB3nzyP6nHPwZVr7X3Zvb27809kMuCwJY6AwTwOHUsEULmnVhwwZwlVXXcW8efNITExkzpw5AMVjJx07diQtLS0w2XmNMUH5AZzANqA9EAGsBrqVqnMX8Jr9ehzwof26m13fAyTZ7TjLaP8A0PZ0fenXr5+pjuEfDzePLn60yvunPPAbs3X4iJPK1n+317z8q3kmIy2nWn1TqjbbsGFDqLtQa9x9991m2rRpoe5Gmcr6nIBkU853ajCvOAYAW40x240x+cBMoPQ10mjgxD2gWcAwERG7fKYxJs8YswPYardX0jBgmzFmF0GS/XhTjj7VjkhnBLO3zeZfW/5VpXYkKpL8XbvY2KMnW0eMwOTn47an6L732FJevXsBS2b9FMiuK6VqEU2rXnmtgD0l3qcAA8urY4wpFJF0IMEu/77UvqVHdMYBZY8qASJyO3A7QJs2barQfdhtmtLVt4ffdryGO1dOZeORjVVqJ2HCBFwJjcldv57j331HUWYmbc5MYMBlSRQW+Niy7ACpuzOr1LZSqvbTtOq1gIhEAJcDH5dXxxjzhjGmvzGmf5MmTap0nI3dHwTgnLguNI9u7tdT5CV5OnWi6QP30/DikVbfcnPxRLo465IkBl/RgfgW0TpFVylVZwQzcOwFWpd4n2iXlVlHRFxALJBWiX0vBlYaYw4GuM8ncURYsy72zX4Sj8PDp1s/Zf3h9VVuT+zsuakvvsjBP/+ZvO3bAXC5HaSn5rD4wy0s/nALezYdqX7nlVIqSIIZOJYDnUQkyb5CGAfMLlVnNjDBfj0WmG8PyswGxtmzrpKATsAPJfa7lgpuUwVK8/bdAWiVupj+jayZGO9ver/K7Xk6dcLVpAmZC77lyDvvkv7JJ9ZxOsRifIbNyw6wduFeVny5s9p9V0qpYAnaGIc9ZnEPMAdrBtRbxpj1IvIU1mj9bOBNYLqIbAWOYAUX7HofARuAQuBuY0wRgIhEAxcBvwpW308Y2KsH/1n9OKO3P8kTPX7Fj8c2+7U+R2neLp3ptHgRAJsHDMSXYz0c1Hd4W/oObwvA7BdXkZ9TWP3OK6VUkAR1jMMY84UxprMxpoMxZrJdNskOGhhjco0xVxljOhpjBhhjtpfYd7K9XxdjzJclyo8bYxKMMenB7PsJDrd1u+r4oW14nB62HN3idwqSMtv1eik8dIi8bdsoOPjzHTeX20FediFH9h/nyP7j5B4vqPaxlFLhm1Z9/PjxdOnShTPPPJNbbrmFgoLqf6fUycHxmuSIslKiR394FbGuKHZl7OKRxac8y+g3Z2xDMufOZfsll7L1/KHk2zlqPFEujh3M5oMnl/HBk8uYMen7E8+tKKWqIVzTqo8fP55Nmzaxdu1acnJyApLfSgPHaQwdPoYvHecD8Hi3WwBIzU6tdrut/vpXWj3/FxLuvAOAwlSrzUFXdGD4bd0Zflt3OvVvSu7xAnxFGjiUqqpwTqsOMGrUKEQEEWHAgAGkpKRU+ndXHk1yeBrRkR4y2l8CWxeS6G7ABa0vYG9W9dMSezp2tJIerlhB2quv4cu1pvpGx3ro1L8ZAMeP5fFT8iEKC3w4XRrjVd22+KMtHN7jf+qeijRuHcOQqztXWCec06qXVFBQ4HcalfLot1ElnBjnOPb2OLziYnfGbv6S/JeAtH1iiu6hqVPZdcON7Ln7HnzHjwPWeAfAf19axSd/Wcknf1nJjtXVv9pRKtxoWnW46667OO+88wJyq02vOCqhU+9zyN7gIS7/ACMancmPaWv5cPOHPNj/wWq37enQnphfDMOXnkFRejrZy5eTt307kT160KpLPK3PiKeo0LpVdXBnBttWppLUq2oPNCoVSqe7MgiGcE+rfsKTTz5Jamoqr7/+evVO0qZXHJXQu0sH/tt1CgDDGvXgsvaXkVeUF5BBa0dkJK1ffpm209+l2URr0N3Yf9nim0dz+a/7MObBvox5sC8NE7y6brlSfgj3tOoA06ZNY86cOXzwwQc4HIH5ytcrjko6cbvqwP9m4O3SBZ/x8e2ebzm/9fk4JDAfxonbVlmLv6Pg0CHE6SLmvCE4oqIAcEU4SU/N4afl9vRdgcQu8UQ2iAjI8ZWqj/xNq37zzTfz7LPPFo9HVFbJtOqtW7cuM616ampqmWnV8/KsMc6nn36azp1PvjI7kVY9PT0dY8xJadVLjnEMGTKETZs2kZWVRWJiIm+++SYjRozgjjvuoG3btgwePBiAK6+8sngcpaokHKZ69u/f31R3nvXnCxZxycLLAPj3hQ/y+A4rTdZ7o96jV5Ne1e4jQH5KCtt+cdFJZc3+8BiNxo+3+vDKGnauOXzS9h7nt+K8a7sE5PhKBdrGjRsrHFMIJ/fccw99+vQJzHoYAVbW5yQiK4wx/cuqr1cclTRq6BCm7f07t229m5GR7Wk49K888O0DZOYHLqttRGIiHRcuxHc8C1NYyI7LR+PL/Ln9Ebd1J/PIzw8fzn5hFXn6lLlStV6/fv2Ijo7mL38JzKSaUNPAUUkiQlxiZ9gKhfnZtG5g5WAMxFPkJbmbNQWaWuMnTmdxWhKwblXFN/95wCwi0qVZdZWqA+pbWnUNHH6I8MYA4Pr6D+QW/B6AR797lPS8dP6v8/8F9FgigiMykiNvvcVRe9ZFRGIi7T7+CHFai0C5IpzsXH2Yab9ZVLyfJ8rFVRPPwhvtDmh/lKoqYwzW+myqNqrKcIXOqvLDOWd24D/N7sZpiojb+xP39L4HQVh7eG1Qjtfs0UeJGzeO2NGj8XbuTO6GDfiys4u3D7g0iTPPb0WXgc3pMrA5LTrEknE4l8y0wF4FKVVVXq+XtLQ0TZtTSxljSEtLw+v1+rWfXnH4ISHGQ+crHiHjtfeQghx+1etX/Gfbf6qVMbcicVeOAazUBEc/+IDs5GR8OTk47bQCbc9MoO2ZCcX1d69PY+faNL19pWqNxMREUlJSSE3VB1drK6/XS2Jiol/7aODwk9ftJNdE4NibzMEfZuFxetiQtoGZm2YyrmvZOWQC4cRU3cOvvoozxgoc7latiB/389OlrgjrAnLttyknzb5yOIUzz29FdKwnaP1Tqixut5ukpKRQd0MFmAYOPzVt4GGpqwsX5i8h85tH6XH+OGZvm83kZZMZ0W4E8d74oBzX06kjjpgY0mf9CwDj80FREQ0vGVV8BdKwcRSRDdxs+/HQzzsa8BUZvNFueg1rXVbTSinlFw0cfor2uPjFpM/58MlruKzofzx1zlP0adqHSf+bRE5hDvEEJ3BE9uhBl+Tlxe+PzpzJgSeexJf9862rmHgPtzx78pOuRYU+XrvnWwry9IlzpVRgaOCooiKHB5cvl5xjh/A4rVtAKZkpNItqhtPhDPrxT9y6KtizG4rsZzlEcDVvftIMFodTEIHszPyTngEBcDiEqNgInfGilPKLBo4qKnA3ICI3H9/fOtNg7N8BuHXurVzX9TomDpwY9OM7G1pXGbuuv+Gk8sZ33UWT++4tfi8iRES6WLsghbULTs3Df+GNZ3DG2S2C21mlVL2igaOKBl79Oz751DAmYwaD3Ak8c94zPL/iefYf318jx48eMoSWzz2Hyfv5KuLQs89RsP/U419ydy+OHjh+SvmC6ZvIOqpTd5VS/tHAUUVd27fl+7YjYe0M3L4CLk66jPc2vkdeUV6NHN8REUHspZecVJb2j2knBZITWnSIpUWH2FPKF76/mcJ8nbqrlPKPBo5qcERYGXP3fPI4UWlpeJ1e1h1ex/0L7ueRAY/QPLp5jfZHvF6OL/uB3bffflJ53BVX0HDUqFPqu9wOfko+yOGUU1M0uz0Ozr+2i2beVUqdQp8cr4Yzu/dgufss4vL2UbTiHUa0G0Gz6GbM2z2PlQdX1nh/Yi+9BHfLlhQdPVb8k528gmP//qTM+l3PbkFkjJvcrPyTfjKP5LJtZSoHd2TU8BkopeoCveKohr7tm3Pk/i9InvILehTlcnWXqzm31bmM+NeIGrtlVVLCbbeRcNttJ5XtuuHG4oWhSitvRba0vVnM/OMP+gS6UqpMGjiqyet2kEsEzpwDpKz4gsiuVvr6dYfX0S62HX2a9glp/8TrpWD3bjLnLzip3BHpJWrAgOKEiSWdeAL9wPZ0nO6yL0obNvaS0DIm8B1WStV6Gjiqyetyku6IJ75gOfGfXUte9Ay8Ti8fbfmIj7Z8xFf/9xWtYlqFrH+uxo05vngxKXfddcq21tOmEXPuOaeUe6PdOBzC6nl7WD1vT5ntemPc3Ppc9Re9V0rVPUENHCIyEngBcALTjDFTSm33AO8C/YA04BpjzE5720TgVqAIuM8YM8cujwOmAWcCBrjFGLM0mOdREYdDuODX/+Dd/8zgxl2PEpGbyWdjPmPOzjk8l/wc6XnpIQ0czR+fRLy9guAJBSkp7L3/forSj5W5jyfKzfg/DiI3q6DM7esW7WXT//ZrumylwlTQAoeIOIG/AxcBKcByEZltjNlQotqtwFFjTEcRGQdMBa4RkW7AOKA70BL4RkQ6G2OKsALRV8aYsSISAUQF6xwqq1mjOHwtesMuKMzLpnl0czrFdQIIyVhHSQ6vl8gzu59U5oyzpuaa3PL71jAhkoYJkeVsO4Kxc2A5XRo4lAo3wbziGABsNcZsBxCRmcBooGTgGA08Yb+eBbws1n9hRwMzjTF5wA4R2QoMEJENwHnATQDGmHwgP4jnUGkuj7UyX96Xj3Hkf2/jHfc8APfNv48W0S2YPmp6cWqSUHPYufcPTplC6gsvnLI9ZuhQWjz1ZLn7nxgDeffR/1HRBUdEpIsxD/bVKb1K1TPBDBytgJI3yFOAgeXVMcYUikg6kGCXf19q31ZADpAK/FNEegErgF8bY055LFpEbgduB2jTpk0gzqdCQ3t14uvNv6TJgUX0Tl9DbHRLbux2IxvSNpB8MJnU7FQSG/iX8z5YnAkJNL73HgoPHDhlW/YPyzm+ZEmF+7fv3YT0Qzn4isqfdZV1LJ/d69M4dihHA4dS9UxdGxx3AX2Be40xy0TkBeAR4A+lKxpj3gDeAOjfv3/Qlx9LbBRN4h3P8dJzk+idtRmvz8dDZz3EVzu+IvlgcshvWZUkIjS5++4yt+2f9DiZ8+dXuH/DxpGcf12XCuvs++kou9enUVigWXmVqm8qFThEJBrIMcb4RKQz0BX40hhT9uipZS9QcgGIRLusrDopIuICYrEGycvbNwVIMcYss8tnYQWO2sNl3QZa9fFkEntfhLe5tULfB5s+oF3DdlzT9Rrcjtq7Hrh4PfgyMjj86qvlVBAaXnopEadZMczptqb5bl56gIPb00973NgmUXQ6q5nf/VVK1bzKXnEsAoaISDwwF1gOXAOMr2Cf5UAnEUnC+tIfB1xXqs5sYAKwFBgLzDfGGBGZDbwvIs9jDY53An4wxhSJyB4R6WKM2QwM4+Qxk5DzNOtM3lE3vfd+QO6B2bS+dzFep5cPN38IQLeEbvRt1jfEvSyfp1MnTH4+qS+8WG6dovQMmj38uwrbadDIS4TXyeZlp94OK5NA+75NcDo1mYFStV1lA4cYY7JF5FbgFWPMMyKyqqId7DGLe4A5WNNx3zLGrBeRp4BkY8xs4E1guj34fQQruGDX+wgrKBQCd9szqgDuBWbYM6q2Azf7c8LBdvu1YyksGM1bf7qdm4o+pUNse76/7ntWp65mwlcTyC7MDnUXKxR/1VXEjRlT7vafzh+KL/v05xDVMILbnj8PY05/l3DVvD0s/fc2ivJ9OCM1cChV21U6cIjIYKwrjFvtstOuVmSM+QL4olTZpBKvc4Grytl3MjC5jPJVQP9K9jskXG43BU4vjkIfxw7tITahGVFua9ZwWk4a6XnpxHpOzVZbW4ir/L8WDo8HX2YGhUePlr+/240zJgZxCMLpp+u6I6y/SlnH8oisYMC9NKfLQYS3rg3TKVX3VfZf3f3AROAT+2qgPbCg4l3CW5E7Bgoh7tUeHI1sQ/RdVvx8bMljADx19lOM6VT+/+xrK0dMDBlffEnGF19WUMlB2xnvEdWnculWIiKtv4YfPLnsNDVLH0a47smBxDYJ+aM8SoWVSgUOY8xCYCGAiDiAw8aY+4LZsbpuyP/dw9IVTfFt/IxBOeuIi0nk2fOe5UjuEaYun8qezLJTedR2LZ7+Izlr1pa7vehIGodfeZWCvfugkoEjqVdjho7vQlFh5Se/pR/KZs2CFLKO5GngUKqGVXZW1fvAHVjpP5YDDUXkBWPMs8HsXF3Wo2Nb6PgQb/1pB+fkrwZfISOTRgLwwsoXyC2qmyvvRfbsSWTPnuVuL9i7l8OvvFrmglLlifC66D7Ev7QsB7ans2ZBimbwVSoEKnurqpsxJkNExgNfYk2BXQFo4DiNQqc9Pff50cQktKTjhFfxurws2L2AA8cP0LZhW37d99ch7mXgiP1U+pEZM8hauOi09RuOGkXDkSP8Po7LHhdJ/mIHG5fs83t/gKTeTegysGYX21KqPqhs4HCLiBu4AnjZGFMgIkF/qK4+aNj5XLaunU/zrE00P74EjjzE8LbDST6YzIqDK/h619fc0euOWpOOpLqccXFEnzeEwv37yd+xvcK6+Sl7KUpPr1LgaNjYS4uOseRlF5Kf6/9Mtcy0XDKP5GrgUKoKKhs4Xgd2AquBRSLSFtDl4Sph3JgrYcyV/OMfL/HLvY9BQQ6/H/R7AKZvmM4zy58htzC33gQOcTpp88Yblaq7+5Zb8R0/JVtMpUR4XVz5235V2hfgy9fWcuxQ7Z4arVRtVdnB8ReBkk+E7RKRC4LTpfpJ7Om421d+DSn7aN97KF77KfOl+5cS54mjY1xHGkc2DmU3a5R4vRSmpHB8aeWz4rtbtyEisfpp6l0RDnKPF7Bn05Fqt9UksQHemNqbDUCpQKvs4Hgs8DhWZlqwZlg9BZw+l4QCwN2wCQDtk58G4Gjq/TQ60/p1PrTwIQDOan4Wb414KzQdDAFXQiOy5u9m9823VHofd5s2dJw7p9rHjoyJIDs9n9l/W1Xttjr2a8qIX55Z7XaUqisqe6vqLWAdcLX9/gbgn8CVwehUfXTNpaNY1+5zdqQc4LwV91KQeZgL2lzA+6PeJ9+Xz0s/vsTR3PIfqquPmj78CLGjR1e6/pH3Zpw2c29lDbyiPe37NrGWAquGRTO3kFPOgldK1VeVDRwdjDH/V+L9k6dLOaJO5o1wcWbfc8mKSyNrRSTu/Bwc4qBHkx4ANItqxqHsQyHuZc1yxkQT1b/ySQCyFi4kKzcw05jdEU5adoyrdjuRDdwU5msGYBVeKhs4ckTkXGPMdwAicg7W2hjKT5FuJ7kmgrjtn5PyZFec0Y1occ9XRLoi2Ze1j0v+fQkAl7a/lDt73xni3tYu4vViCgrYOsL/WVjidtNyytRTVkOsLleEk31bj/HeHwK7enGvYa3pMbR2rN+iVGmVDRx3AO/aYx0AR7Gy2io/dWnegC/a3szR9B9wZO6jb9Z6OLqD0R1Hk1uUizGG5APJLExZqIGjlIYjRlCwezfGj3xWACY/n8y5c8lZszrggePM81vh9pw2bZtfdm9IY/f6NA0cqtaq7Kyq1UAvEWlov88QkfuBNUHsW73kdTu58taHAXjx9dfou/9hKMilT4uB9Glqpeh4YMED7MzYGcJe1k6ejh1pOXWq3/sVZWWROXduhWusV1Xb7gm07Z4Q0Db/9cwKfSJe1Wp+pRY1xpR8duM3wN8C2pswIxGRAGyc8wbpUV/RKak9CWffgMfl4XDOYd5Z/w4ALaJbMLzd8FB2tU5zeKxnZI5/v5QKF0mvTFuRXmKvvBJHRPCWw3VFOMg4nMOqb3YHrM3GrRuQ2CU+YO2p8FadnNTV+xeoaNA0ibydbs7YO8sq+AnofgHtGrbj8+2f81zyc8V1l7ZcSkxETGg6WseJ2407MZHjixZzfNHiarfnbtWKmCFDAtCzssU1iyJl01GWzNoasDZjGnmY8KdzAtaeCm/VCRyacqSabrrkPLKG7iS/MJ+X/v48j+a/BHlZ3NHrDm7odgPGGD7Z+gnPLH+GnMIcDRzV0OHLL/DlVe9WVf62bey8ZlyVn3avrPPGdWbwFR0C1t6Sf29l24rwmrGngqvCwCEimZQdIASIDEqPwkxMtBUM8t2xkA95mWn4ojOJdLtwREQWL/h0LO9YceCIdOmv3l/iduN0V+/pbme8daunKCMDX07gJxWK14uIICLFa5QEgifSRWG+j4IATxt2uR1INW/9qbqpwr+dxpgGNdWRcGciYuA4eKaPAsCHA677kGhPNABXzv75Wctf9vgl9/XV5VBqmiPa+iwOTHqcA5MeD3j7DUeNotXzfwl4uxFeJ0WFPt64b2FA2+08sBkX3RzYWWqqbtB1N2uJMZdfyeKlx3AXZrP9QBrXZc+AtK0M7n8Tjwx4hLwi6zbL9A3T2ZG+I8S9DU+uhARaPjOVwkOBv+1z7NNPydtecTbhqjrjnJY43U6ML3B3lzct3c/R/ZokMlxp4Kgleic1g6SJAMz9ZAWsngGFOUS5oxh/xvjienN3ziWnSJ+9DJXYyy8PSru5GzaQu2FjUNqOjvXQ56I2AW3z0M4MjhzQwBGuNHDUQq6ISHxGOPDdDPYmL8PpEDr1GEiDC3+Dx+lhU9omfv/d74vrD0kcwsh2I0PYY1Vd4vFScOgQ+x5+pEaPGzVgAHH/53/KOWeEg6wjuXzz9oYK6zVvH8uZ51U/m7GqXTRw1EIDkhL4buUAOubvoFXej8SYLKIXfw0X/oZzW53LweyDrDi4AoC0nDR+OvqTBo46LnrQQLKXLyc7ObnGjll49CjZK1ZUKXAkdmnEgW3p7PvpWLl1crMK2LPhiAaOekiMqf+zavv372+Sa/AfZCAdyszl3Sn38Fv3x/CHw+A8eWbQg98+yE/HfmL2FbND1ENVV+3/wyQyv11A58XVf7alLAtmbGLn6sPc/My5QWlfBZeIrDDGlJmFVK84ajmv20ku1lPKW374Cp8rkgiXg6T2nZHYRLwuL1n5Waw6tOrnfVxeusR30amSqkLi9WKOZ5P9448Bac+TlIQzLu6ksvr/39LwpIGjlotyO8l0NASg85zri8vzvY2JeGQbsZ5YUnNSueHLG07a7/VfvM7Zrc6u0b6qusUZH4cvO5td114XkPaizx5Mm7d+XohMAMLgjkY4CmrgEJGRwAuAE5hmjJlSarsHeBfoB6QB1xhjdtrbJgK3AkXAfcaYOXb5TiDTLi8s71KqvnA5Hdx650Os3nY2UlTA0ewCji59h9G5S8EY7up1F+e2PBdj/9/uUPYhJv1vEmm5aSHuuartEm66ichevSAA03RTX3qJovSM01dU9ULQAoeIOIG/AxcBKcByEZltjCk5DeNW4KgxpqOIjAOmAteISDdgHNAdaAl8IyKdjTEnHn29wBhzOFh9r206t0yAltY00H3HcnhvyTzEaaAon5iImJOuLA4cPwBQ/NyHUuVxREURc05g8lcd/eADCvbvP7lQb5XWW8G84hgAbDXGbAcQkZnAaKBk4BgNPGG/ngW8LNaN+dHATGNMHrBDRLba7QV2tZw6yOt2kmePeez401n4xAFAg6hIml7zMt7GVo6jV1e/ysxNM4v38zg9/PHcP9I+tn3Nd1rVfw4Bn6aCDxeOILbdCthT4n2KXVZmHWNMIZAOJJxmXwPMFZEVInJ7eQcXkdtFJFlEklNTU6t1IrVJfJSbuF6XkBx9Hse8rcjwtGCfL56mmRtg1xJiPbFcf8b1dE/oTsuYlrSMaUkjbyPWHF7DusPrQt19VU+Jwwm+k3NhCTrEUV/VxcHxc40xe0WkKfC1iGwyxiwqXckY8wbwBljTcWu6k8EiItx79ShgVHHZne98z5AdI6AwFxHh4QEPn7TPoexDDPt4GLmFgVmvW6lTOB1+r8yo6q5gBo69QOsS7xPtsrLqpIiIC4jFGiQvd19jzIk/D4nIJ1i3sE4JHOHEHeGhECcH135LysGf7yu3bdKQ5oOuxuO0FjJatn/ZKftGuiK5OOliXI66+H8IVVuIwwlFpbLvCjoft54K5rfFcqCTiCRhfemPA0rP+5uNtXb5UmAsMN8YY0RkNvC+iDyPNTjeCfhBRKIBhzEm0349HHgqiOdQJ7RuFMkuX1M6HF5Cq8NLTt7oLSR64K+I98Qzd9dc5u6ae8r+LWNa0q9ZvxrqraqXHA6MjnGEjaAFDmNMoYjcA8zBmo77ljFmvYg8BSQbY2YDbwLT7cHvI1jBBbveR1gD6YXA3caYIhFpBnxiP9jmAt43xnwVrHOoK347vAupA77ncN7P0yEf/89a/r7/OsjLxOVwMWfsHI4XnLwA0ca0jdw1765TypXylzgcp1xxWBcceslRHwX1/oQx5gvgi1Jlk0q8zgWuKmffycDkUmXbgV6B72ndJiI0jY8D4orLfNGp5OMiosDKpBvpijxlAahm0c0AdOxDVZ/TqVccYURvbNdTUREujhsvEUv+Bkv+dtK23Ih4vA/8SJQrCoAHFz4IZazx069ZP94e+XbQ+6rqPnE6Tp2OK6JjHPWUBo566lfnt2exbxJNjm85qfzYrnVcnP89ZOyjVdNuPDrwUY7kHjll///t+x+bjmyqqe6qus6hVxzhRANHPdW5WQM6j/vVKeUP/HEKFxd9DwXW1N1ru15b5v6FvkI2HK54rQWlijmk7FlVql7SwBFmfC4vFMGGjx4n051w0rYYj4vuw8ZDhwvxOD0UmkKeWvoUUs43wMikkZzV/Kya6Laq5cThpOjo0VB3Q9UQDRxhpllSD/Zsak6zjDU0K1FuDMSSRcGiVNwdLqRH4x40jWzKvN3zymwnPS+dQzmHNHAoAIoyTk1wqE+O118aOMLMo+OGAZtPKZ+xbBdt/3stAwqsdaTPaXUO864uO2gAjP9ivM7GUsU87ZMAMIWFiMv+WtFbVfWWBg4FgNdlLRhVkJVGyrpTnzAHaJkQi7d5FxDB6/SSnpfO5iOnBqET3A43SbFJuqBUGBCPFwBfbh7OmBJfK3rJUS9p4FAAxEW5OUYM0Rk/0n7W8PIrXjsTulxMrCeWHw78wNjPxlbY7pQhU7ik/SUB7q2qdez/HOSuX0/0wAFWkV5y1FsaOBQA53duwvdjn+XHAz+Uuf3bVVt4IOdlyLTW+3hkwCOMShpVZl2A3KJcJi6eyOGcsFk2JaxF9rKey/VlZZ5Urtcb9ZMGDgVYKw2e2+sM6HVGmdun7fkGUl6GQmuBqKZRTflF21+U216Br4CJiyfqOEiYcDZsAIAvt8TnrRcc9ZYGDlUpDreVriR17nOkffN22ZUEmjRpTsJN7+OOiMIpTj7a8hGL9y4+bfuJDRL587l/1vGQOkq81hhH6gsvcnT6ewBkeAfh83Rh57iynxVSweeMjaX1668FvF0NHKpSLunbnq9TL6d5QUq5dZw5aSTsWwBHtkPzM7mx242Vevp8//H9rE5dze8H/p4GEQ0C2W1VQ9zNmxM7ejSFJRZNE6cLEBxRUaHrWJiTqMjTV6oCDRyqUkb2aAE9pldY54E/TuWvRX8C+/bUb/r/plJtf7jpQ55e9jR5RXk0QANHXSQuFy2nTjmpbPesn5DF+2jz+psh6pUKFg0cKmB8rkgogk0LPyQzpuxB9hMaN4wkadAVEBmPx2UtNPX59s9p5G1UqWO5HW7Ob33+KRl/Ve2hNx3rLw0cKmCkYQtIha4/vVG5HRypcP7vaB7dHIDnkp/z63hPnf0UYzqN8bebqibpcxz1kgYOFTBTbr+SPQfOwnGahaFmrUjh9nXX481NR4BBLQbxzdhvyC/Kr9RxsgqyuPq/V5NVkBWAXqug0YkO9ZYGDhUwXreT1q3bnbaee7eHHCJw52UX/wU8sahUZeQV5Z30p6rF9IKjXtLAoWpclNtJNl4arnib7BUzKr1ffnRz4n6znAhnBA5x8Pcf/84bayp5W6yEBu4GfHjZhzSObOz3vqryrKVjVX2kgUPVuFE9WrBk129pkbG20vv4Dqzj7OzVkJ2GNGzJE4OfYHv6dr+PvTdrL1/v+pq9WXs1cChVRRo4VI1r2tDLmOvu9GufZ595wgoc9hrqVR0UX35gOV/v+rrS4ymqGvSSo97SwKHqBJ897Xb1+4+R44r1e/8Il5OeF12PJzoagA82fcDCPWUstO4HhzgY02kMSbFJ1WqnPisq9LFk1k/VbiepdxNadoyrfodUQGjgUHVCg8RupK1vSKfD5a8RUh4DRJJP5rx9tLpuGk2jmvLd3u+q3aecwhwKTSG/O+t31W6rPsrPtZaSXbMgBYfLUeV2CvOLOLL/OC3v7R2gnqnq0sCh6oS7rr4MuKxK+y7dlobznYvpkp9NQmQC867yP/iU5fwPz9ckjhUozLcCx5BrOnPmea2q3M4nf1lJYb4vUN1SAaCBQ9V7kRFOMkwEvpxjpGxdF5A2WyS2xev0ciT3CLszdgekzZIc4qBVTKs6nfTxxJe9K6LqVxvW/k6OH8vl2KHsQHQrrDgcQsPGgc+uoIFD1XsNvS5SiCI+Yxnx750TkDYPRbanwRkdmLd7XrnrslfXxAETue6M64LSdk3YtvIQABGe6n3NeKJc7F5/nBmTvg9Et8JKZMMIbnnm3IC3G9TAISIjgRcAJzDNGDOl1HYP8C7QD0gDrjHG7LS3TQRuBYqA+4wxc0rs5wSSgb3GmEuDeQ6q7ktqHM2BK54hee/SgLRX8ONMeuVtZcqQj9l4ZGNA2ixt0pJJHMo+FJS2a1rr7pXLP1aes6/sQNszEwLUm/DirMbYUkWCFjjsL/e/AxcBKcByEZltjNlQotqtwFFjTEcRGQdMBa4RkW7AOKA70BL4RkQ6G2OK7P1+DWwEGgar/6r+EBHO7tcb+vUOSHsfr19O/7z1dIzvSMf4jgFps7Q/L/szuUX1Y/zE6aze7baYeC9dBjYPUG9UIATzimMAsNUYsx1ARGYCo4GSgWM08IT9ehbwslg3dUcDM40xecAOEdlqt7dURBKBS4DJQOXydisVQD6nBzcFrP3z0KAdw9ksi883fcq2Y9uCdozyjOk4hlHty18W2G91eJxGlS2YgaMVsKfE+xRgYHl1jDGFIpIOJNjl35fa98S0jL8Bv4OKF24QkduB2wHatGlTpRNQqiwNeoxkw8o1uH3By5U1JjOTRZGuGp+1teXoFlwOV0ADh8aN+qdODY6LyKXAIWPMChEZWlFdY8wbwBsA/fv31+dXVcCMung0XDw6qMeImHIO56c76Hd7xYtnBdqELycEPHlkXZ4ZpsoWzMCxF2hd4n2iXVZWnRQRcQGxWIPk5e17OXC5iIwCvEBDEXnPGHN9cE5BqdAocnpo5NvPyq/ertHj5h9LI7Uog9c/fRSArIad8EXE0LV5A7xuZ3E9hzgY1GIQ0e7oGu2fqh2CGTiWA51EJAnrS38cUHpu4WxgArAUGAvMN8YYEZkNvC8iz2MNjncCfjDGLAUmAthXHL/VoKHqo4KopnQ6voKk739do8ft0LgR6xrE8HL6Z1ZBur2hjKGWB/o9wC1n3lJuW10GNmfzsgOB76QKuaAFDnvM4h5gDtZ03LeMMetF5Ckg2RgzG3gTmG4Pfh/BCi7Y9T7CGkgvBO4uMaNKqXqv821vsWPrOmo6S+ANvgKG5R4EoNl3k8hL6Mq1Oy/h8t6tuG2IlZPLGMPYz8aSV1jxLa0Lb+zKedd2DnqfVc0L6hiHMeYL4ItSZZNKvM4Fripn38lYM6fKa/tb4NtA9FOp2sbliSKp+4CQHLvLiRfJz4M7Ai+JeE1LOsdbQcDYy8Ga0wQ1h9NBhDM4zxGo0KpTg+NKqRrkjoKfvma+7zvMSjjyY4ltSXH8Z+EyPpnfnU/uPoeYaj4druoW/bSVUmU77yHY8hXHDh8nNfPnacHWNOT1JHqOM29nFocz8zRwhBn9tJVSZes8HDoPpz3QvmR55kH49y9oEesNUcdUqOkNSKWUUn7RwKGU8puYn4fG9ena8KOBQylVJaIhI2xp4FBK+U3QK41wpoFDKeWfUrmnTjzXocKHBg6llN80bWF408ChlPKbBo7wpoFDKVUlp0s5ouovDRxKqWrR8BF+NHAopfykN6rCnQYOpZTfrOm4eq0RrjRwKKWqRWfjhh8NHEopv4kGi7CmgUMp5R8RTTcS5jStulLKb3kOB//NX0d0xymM/s+U4nKnQ/8vWpu4pQErbvlvwNvVwKGUqjJfQRwmv1Hx+8SE6BD2RpUW6QrO56GBQylVZflp51OUdUbx+zm/viSEvVE1RQOHUqrKLnYsI9H5088F320JXWfUqdzRMPD2gDergUMp5aefHwB8rGgezd1FP2/6JgTdUeWLbqqBQylVu5x//E3+OLo7Y/q0wuUQRPSp8nCggUMpVWV5ROCMiMLt1UHxcKJz55RSVRJfZN2iatbQG+KeqJoW1MAhIiNFZLOIbBWRR8rY7hGRD+3ty0SkXYltE+3yzSIywi7zisgPIrJaRNaLyJPB7L9Sqmzv7jvAv/bu5x839ufcTo1D3R1Vw4IWOETECfwduBjoBlwrIt1KVbsVOGqM6Qj8FZhq79sNGAd0B0YCr9jt5QEXGmN6Ab2BkSIyKFjnoJQqgwh98vJpUuTjom7NQt0bFQLBvOIYAGw1xmw3xuQDM4HRpeqMBt6xX88Chok1ujYamGmMyTPG7AC2AgOMJcuu77Z/NPeBUqHgcIe6BypEghk4WgF7SrxPscvKrGOMKQTSgYSK9hURp4isAg4BXxtjlpV1cBG5XUSSRSQ5NTW1+mejlDqZyxPqHqgQqXOD48aYImNMbyARGCAiZ5ZT7w1jTH9jTP8mTZrUaB+VCgsOZ6h7oEIkmIFjL9C6xPtEu6zMOiLiAmKBtMrsa4w5BizAGgNRStUU47P+FA0c4SqYgWM50ElEkkQkAmuwe3apOrOBCfbrscB8Y4yxy8fZs66SgE7ADyLSRETiAEQkErgI2BTEc1BKleaznxTXK46wFbQHAI0xhSJyDzAHcAJvGWPWi8hTQLIxZjbwJjBdRLYCR7CCC3a9j4ANQCFwtzGmSERaAO/YM6wcwEfGmMDnDFZKlc+cCBz6/HC4Cuonb4z5AviiVNmkEq9zgavK2XcyMLlU2RqgT+B7qpSqNF+h9afeqgpbdW5wXCkVYicWGffEhLYfKmT0WlMp5Z+4NnDBY9CzzJsFKgxo4FBK+UcEzn8o1L1QIaS3qpRSSvlFA4dSSim/aOBQSinlFw0cSiml/KKBQymllF80cCillPKLBg6llFJ+0cChlFLKL2JM/V9AT0RSgV2h7oefGgOHQ92JGqbnHB70nOuGtsaYMhczCovAUReJSLIxpn+o+1GT9JzDg55z3ae3qpRSSvlFA4dSSim/aOCovd4IdQdCQM85POg513E6xqGUUsovesWhlFLKLxo4lFJK+UUDRw0RkdYiskBENojIehH5tV3eSES+FpGf7D/j7XIRkRdFZKuIrBGRviXammDX/0lEJoTqnCpLRJwi8qOI/Nd+nyQiy+xz+1BEIuxyj/1+q729XYk2Jtrlm0VkRIhOpVJEJE5EZonIJhHZKCKD6/vnLCIP2H+v14nIByLirW+fs4i8JSKHRGRdibKAfa4i0k9E1tr7vCgiUrNn6AdjjP7UwA/QAuhrv24AbAG6Ac8Aj9jljwBT7dejgC8BAQYBy+zyRsB2+894+3V8qM/vNOf+G+B94L/2+4+Acfbr14A77dd3Aa/Zr8cBH9qvuwGrAQ+QBGwDnKE+rwrO9x3gNvt1BBBXnz9noBWwA4gs8fneVN8+Z+A8oC+wrkRZwD5X4Ae7rtj7Xhzqcy73dxHqDoTrD/Af4CJgM9DCLmsBbLZfvw5cW6L+Znv7tcDrJcpPqlfbfoBEYB5wIfBf+x/FYcBlbx8MzLFfzwEG269ddj0BJgITS7RZXK+2/QCx9peolCqvt5+zHTj22F+GLvtzHlEfP2egXanAEZDP1d62qUT5SfVq24/eqgoB+9K8D7AMaGaM2W9vOgA0s1+f+Md4QopdVl55bfU34HeAz36fABwzxhTa70v2v/jc7O3pdv26dM5JQCrwT/v23DQRiaYef87GmL3Ac8BuYD/W57aC+v05nxCoz7WV/bp0ea2kgaOGiUgM8C/gfmNMRsltxvqvRr2ZHy0ilwKHjDErQt2XGuTCup3xqjGmD3Ac6xZGsXr4OccDo7GCZksgGhgZ0k6FQH37XCuigaMGiYgbK2jMMMb82y4+KCIt7O0tgEN2+V6gdYndE+2y8spro3OAy0VkJzAT63bVC0CciLjsOiX7X3xu9vZYII26dc4pQIoxZpn9fhZWIKnPn/MvgB3GmFRjTAHwb6zPvj5/zicE6nPda78uXV4raeCoIfYMiTeBjcaY50tsmg2cmFkxAWvs40T5jfbsjEFAun1JPAcYLiLx9v/0httltY4xZqIxJtEY0w5rEHS+MWY8sAAYa1crfc4nfhdj7frGLh9nz8ZJAjphDSTWOsaYA8AeEeliFw0DNlCPP2esW1SDRCTK/nt+4pzr7edcQkA+V3tbhogMsn+HN5Zoq/YJ9SBLuPwA52Jdxq4BVtk/o7Du7c4DfgK+ARrZ9QX4O9bMkrVA/xJt3QJstX9uDvW5VfL8h/LzrKr2WF8IW4GPAY9d7rXfb7W3ty+x/+/t38VmavFsE7uvvYFk+7P+FGv2TL3+nIEngU3AOmA61syoevU5Ax9gjeEUYF1Z3hrIzxXob//+tgEvU2qCRW360ZQjSiml/KK3qpRSSvlFA4dSSim/aOBQSinlFw0cSiml/KKBQymllF80cKiwJSJ/FpELROQKEZno575N7MyuP4rIkFLbpolIN/v1owHu800i0rKsYylVU3Q6rgpbIjIfuAT4EzDLGLPEj33HAb8wxtx2mnpZxpgYP/vlNMYUlbPtW+C3xphkf9pUKpD0ikOFHRF5VkTWAGcBS4HbgFdFZFIZdduJyHx7TYV5ItJGRHpjpdMeLSKrRCSy1D7fikh/EZkCRNp1ZtjbrheRH+yy10XEaZdnichfRGQ1MFhEJonIcrHWt3jDfgJ5LNZDYjNOHPfEsew2rrXXc1gnIlNL9CdLRCaLyGoR+V5EmtnlV9l1V4vIooD/olX9FeonEPVHf0LxgxU0XgLcwJIK6n0GTLBf3wJ8ar++CXi5nH2+xX5SGMgqUX6G3Z7bfv8KcKP92gBXl6jbqMTr6cBlpdsu+R4rueBuoAlWosX5wBUl2j6x/zPAY/brtUAr+3VcqD8T/ak7P3rFocJVX6xFg7oCGyuoNxhrESqwvsDPrcYxhwH9gOUissp+397eVoSVAPOEC+wxlLVYySG7n6bts4BvjZVosBCYgbXwEEA+1hoZYKU7b2e/XgK8LSK/BJxVPSkVflynr6JU/WHfZnobK/voYSDKKpZVWIsG5QTz8MA7xpiyBuJzjT2uISJerKuR/saYPSLyBFZ+p6oqMMacGMwswv53b4y5Q0QGYo3zrBCRfsaYtGocR4UJveJQYcUYs8oY05ufl+6dD4wwxvQuJ2j8DyuzL8B4YLGfhywQK50+WMnwxopIUyher7ptGfucCBKHxVq/ZWyJbZlYSw+X9gNwvog0tsdNrgUWVtQxEelgjFlmjJmEtfhU64rqK3WCXnGosCMiTYCjxhifiHQ1xmyooPq9WKv5PYT15Xqzn4d7A1gjIiuNMeNF5DFgrog4sLKs3g3sKrmDMeaYiPwDK1PqAWB5ic1vA6+JSA7WbbQT++wXkUewUpkL8Lkx5nRpuZ8VkU52/XlYt+6UOi2djquUUsoveqtKKaWUXzRwKKWU8osGDqWUUn7RwKGUUsovGjiUUkr5RQOHUkopv2jgUEop5Zf/B0j+hYOn2isfAAAAAElFTkSuQmCC\\n\",\n      \"text/plain\": [\n       \"<Figure size 432x288 with 1 Axes>\"\n      ]\n     },\n     \"metadata\": {\n      \"needs_background\": \"light\"\n     },\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"history_list = []\\n\",\n    \"for i in [1,2,4,10,12]:\\n\",\n    \"    \\n\",\n    \"    with open(\\\"diff_layers/\\\" + str(i) + '.out') as f:\\n\",\n    \"        lines = f.readlines()\\n\",\n    \"        loss = extract_line(lines, \\\"loss: \\\")\\n\",\n    \"        x1, x2 = 1000,11000\\n\",\n    \"        plt.plot(range(x1, x2), loss[x1:x2], label=\\\"# of layers:\\\"+str(i))\\n\",\n    \"plt.legend()\\n\",\n    \"plt.xlabel(\\\"# of iterations\\\")\\n\",\n    \"plt.ylabel(\\\"Loss\\\")\\n\",\n    \"plt.savefig(\\\"diff_layers\\\", dpi=600)\\n\",\n    \"plt.show()\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"acoustic-floor\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  }\n ],\n \"metadata\": {\n  \"kernelspec\": {\n   \"display_name\": \"Python 3\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.7.11\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n"
  },
  {
    "path": "pretraining/train.py",
    "content": "import numpy as np\nimport _pickle as pickle\nimport pandas as pd\n\nfrom cpt import *\nfrom DataGenerator import *\n\nprint(\"------LOADING DATA------\")\npath = \"../data/seqs/\"\n\npid_seq = pickle.load(open(path+\"pid_seq\",\"rb\"))\nage_seq = pickle.load(open(path+\"age_seq\",\"rb\"))\nsex_seq = pickle.load(open(path+\"sex_seq\",\"rb\"))\n\ncode_seq = pickle.load(open(path+\"code_seq\",\"rb\"))\ndate_seq = pickle.load(open(path+\"date_seq\",\"rb\"))\nutil_seq = pickle.load(open(path+\"type_seq\",\"rb\"))\n\nprint(\"------LOADING DIC------\")\npath = \"../data/cat/\"\n\ndiag2cat = pickle.load(open(path+\"diag2cat\",\"rb\"))\nproc2cat = pickle.load(open(path+\"proc2cat\",\"rb\"))\ndrug2cat = pickle.load(open(path+\"drug2cat\",\"rb\"))\n\nprint(\"# of patients: \", len(pid_seq))\n\nMAX_VISIT=30\nMAX_CODE=10\nMAX_DEMO=2\nPATIENT_DIM=100\n\nBATCH_SIZE = 100\nTRAIN_RATIO = 0.8\nDATA_SIZE = len(age_seq)\nEPOCHS = 1000\n\ncode_feature, code2int = process_code(code_seq)\nutil_feature, util2int = process_util(util_seq)\ndemo_feature, demo2int = process_demo(age_seq, sex_seq)\n\ndate_feature = date_seq\n\ncat_seq = get_cat(code_seq, diag2cat, proc2cat, drug2cat)\ncat_feature, cat2int = process_code(cat_seq)  \n\npickle.dump([code2int, util2int, demo2int, cat2int], open(\"vocabs/vocabs\",\"wb\"))\n\nparams = {\n    'seqs':[demo_feature, code_feature, util_feature, date_feature, cat_feature],\n    'vocab_sizes': [len(code2int), len(cat2int)],\n    'batch_size':100,\n    'max_visit':MAX_VISIT, \n    'max_code':MAX_CODE,\n}\n\nfrom sklearn.model_selection import train_test_split\ntrain_IDs, valid_IDs = train_test_split(range(DATA_SIZE), train_size=TRAIN_RATIO, random_state=42)\ntrain_generator = DataGenerator(list_IDs=train_IDs, shuffle=True, **params)\nvalid_generator = DataGenerator(list_IDs=valid_IDs, shuffle=False, **params)\n\nm = model(\n    patient_dim=PATIENT_DIM,\n    max_visit=MAX_VISIT,\n    max_code=MAX_CODE,\n    max_demo=MAX_DEMO,\n    code_vocab=len(code2int),\n    demo_vocab=len(demo2int),\n    util_vocab=4,\n    date_vocab=366,\n    cat_vocab=len(cat2int),\n)\n\n# m.compile(optimizer='RMSprop', loss=model_losses, metrics=model_metrics, loss_weights=model_weights)\nm.compile(optimizer='RMSprop', loss=model_losses, loss_weights=model_weights)\nprint(m.summary())\n\nearlyStopping = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=5, verbose=0, mode='min', restore_best_weights=True)\nm.fit(\n    train_generator,\n    epochs=EPOCHS,\n    validation_data=valid_generator,\n    verbose=2,\n    callbacks=[earlyStopping],\n)\n\nprint(\"------SAVING MODEL-------\")\nm.save('./saveModel/')\n\n\n\n\n\n\n\n"
  }
]