[
  {
    "path": "README.md",
    "content": "# Criteo 1 TiB benchmark\n\n\n\n## Table of contents\n\n* [Introduction](#introduction)\n* [Task and data](#task-and-data)\n* [Algorithms](#algorithms)\n* [Setup](#setup)\n* [Experiment layout](#experiment-layout)\n  * [Hyperparameter optimization](#hyperparameter-optimization)\n  * [Data format for Spark.ML](#data-format-for-sparkml)\n  * [Code](#code)\n* [Results](#results)\n  * [Local training - Vowpal Wabbit & XGBoost](#local-training---vowpal-wabbit--xgboost)\n  * [Distributed training - Spark.ML](#distributed-training---sparkml)\n  * [Distributed training - Time vs. Cores](#distributed-training---time-vs-cores)\n  * [Comparison of local vs. remote](#comparison-of-local-vs-remote)\n* [Conclusion](#conclusion)\n* [Resources](#resources)\n\n\n## Introduction\n[_(back to toc)_](#table-of-contents)\n\nThis project is a minimal benchmark of applicability of several implementations of machine learning algorithms to training on big data. Our main focus is [Spark.ML](http://spark.apache.org/mllib/) and how it compares to commonly used single-node machine learning tools Vowpal Wabbit and XGBoost in terms of scaling to terabyte (billions of lines) train data. Quick web search shows that many people tested Spark but not on tasks requiring a cluster so they are mostly single-node tests.\n\nThis project is inspired by https://github.com/szilard/benchm-ml but is focused on training models on billions of lines of train data, including Spark.ML in multinode cluster environment.\n\n\n\n## Task and data\n[_(back to toc)_](#table-of-contents)\n\nOur target application is prediction of click-through ratio (CTR) of banners in online advertising. [Criteo released](http://labs.criteo.com/2015/03/criteo-releases-its-new-dataset/) an industry-standard open dataset which represents banner impressions in online advertising during the timespan of 24 days. It is more than 1 terabyte in size and consists of more than 4 billion lines of data. Each line represents a banner impression and contains 40 columns separated by tabulation:\n\n- the first column is a label - {0, 1} - 1 meaning the banner was clicked and 0 otherwise;\n- 13 numeric columns;\n- 26 categorical columns with categories being 32-bit hashes.\n\nThis is how it looks like:\n\n![Dataset schema](images/dataset.png)\n\nAll the data except the last day was concatenated and sampled into training sets of 10ⁿ and 3×10ⁿ lines with `n ∈ {4, 5, ..., 9}` (i.e. train samples' sizes are 10k, 30k, 100k, ..., 1kkk, 3kkk lines). The last day was used for testing - a sample of one million lines was taken from it. All samples were converted to\n\n- LibSVM format for training XGBoost models and as a source for the transformation to Spark.ML DataFrame;\n- Vowpal Wabbit data format.\n\nData for Spark.ML models was processed on-the-fly from LibSVM format:\n\n- into a dataset of tuples of \"label\" (integer) and \"features\" (SparseVector of size 10⁵ using [hashing trick](https://en.wikipedia.org/wiki/Feature_hashing#Feature_vectorization_using_the_hashing_trick) for all features) for Spark.ML LogisticRegression;\n- into a dataset of tuples of \"label\" (integer) and \"features\" (SparseVector of size 39 taken as-is from corresponding columns, see below) for Spark.ML RandomForestClassifier.\n\n\n\n## Algorithms\n[_(back to toc)_](#table-of-contents)\n\nHistorically, we make use of [Vowpal Wabbit](https://github.com/JohnLangford/vowpal_wabbit) and [XGBoost](https://github.com/dmlc/xgboost) exploiting \"Local train + Distributed apply\" scenario. Our task was to run a performance test of our currently used approach and Spark.ML library algorithms.\n\nWe used the following non-distributed algorithms:\n\n- Vowpal Wabbit - it implements logistic regression with a hashing trick and reads the data only once never keeping more than one sample in memory (it is an out-of-core implementation);\n- in-memory XGBoost - gradient-boosted trees implementation that (by default) loads the whole data into memory (which is faster than multiple reads from disk, but we are limited in size by machine memory);\n- out-of-core XGBoost - a variant of XGBoost training which uses an on-disk cache; this is slower (compared to the in-memory variant) but potentially we can train on the data limited in size only by the size of HDD.\n\nSpark.ML contains following classification algorithms:\n\n- [LogisticRegression](http://spark.apache.org/docs/latest/ml-classification-regression.html#logistic-regression),\n- [RandomForestClassifier](http://spark.apache.org/docs/latest/ml-classification-regression.html#random-forest-classifier),\n- [NaiveBayes](http://spark.apache.org/docs/latest/ml-classification-regression.html#naive-bayes),\n- [DecisionTreeClassifier](http://spark.apache.org/docs/latest/ml-classification-regression.html#decision-tree-classifier),\n- [GBTClassifier](http://spark.apache.org/docs/latest/ml-classification-regression.html#gradient-boosted-tree-classifier),\n- [MultilayerPerceptronClassifier](http://spark.apache.org/docs/latest/ml-classification-regression.html#multilayer-perceptron-classifier).\n\nOur preliminary research shows that four of the algorithms are not well-suited for our task of CTR prediction:\n\n- NaiveBayes provides significantly worse logistic loss (which is an essential metric of CTR models' quality) than all other models;\n- DecisionTreeClassifier suffers in quality in comparison to the RandomForestClassifier but still requires roughly the same amount of time to train;\n- GBTClassifier (Spark.ML implementation of gradient-boosted trees) and MultilayerPerceptronClassifier do not support prediction of probabilities that are required by the task (these two models are not shown on graphs above).\n\n![ROC AUC](images/roc_auc.cluster_selection.png) ![Log loss](images/log_loss.cluster_selection.png) ![Training time](images/train_time.cluster_selection.png)\n\nThus we use only LogisticRegression and RandomForestClassifier for our testing purposes.\n\n\n\n## Setup\n[_(back to toc)_](#table-of-contents)\n\nLocal models were trained on a 12-core (24-thread) machine with 128 GiB of memory. Distributed training was performed on our production cluster (total capacity is approximately 2000 cores and 10 TiB of memory); for the experiment a small part of resources has been allocated - 256 cores and 1 TiB of memory for training on datasets upto 300 million of lines and 512 cores and 2 TiB of memory for training on one billion and 3 billion lines of train data. 4 cores and 16 TiB of memory per Spark executor was used.\n\nFor the experiment we used Vowpal Wabbit 8.3.0, XGBoost 0.4 and Spark 2.1.0 running on a Hadoop 2.6 cluster (using YARN as a cluster manager).\n\n\n\n## Experiment layout\n\n### Hyperparameter optimization\n[_(back to toc)_](#table-of-contents)\n\nOur first idea was to skip models' hyperparameters optimization completely, but unfortunately XGBoost's default hyperparameters are not good enough for training even on million lines of data - the default number of trees is only 10, and it hits the ceiling quite soon:\n\n![ROC AUC](images/roc_auc.why_optimize.png) ![Log loss](images/log_loss.why_optimize.png)\n\nThese figures reminded us that production usage of any machine learning model is associated with optimization of its hyperparameters, and in our experiment we should do the same. For optimization of models' hyperparameters (including Spark.ML ones) we used the million-line sample of train data and 5-fold cross validation for metric (log loss) averaging.\n\n\n\n### Data format for Spark.ML\n[_(back to toc)_](#table-of-contents)\n\nWe tried to use [one-hot-encoding](https://www.quora.com/What-is-one-hot-encoding-and-when-is-it-used-in-data-science) of categorical features, but due to very large number of unique values it turned out to be very time and memory consuming, so for Spark.ML we decided to try the hashing trick. Spark.ML LogisticRegression was trained using this approach. We sticked to hashing space of 10⁵ hashes as it turned out to give about the same quality as VW on large samples. Taking less hashes usually leads to better quality on smaller data (because of less overfitting) and worse quality on bigger data (because some patterns in data are consumed by collisions in hashing space):\n\n![ROC AUC](images/roc_auc.lr_hash_size.png) ![Log loss](images/log_loss.lr_hash_size.png)\n\nRandomForestClassifier was very slow to train even with a thousand hashes, so we used \"as-is\" format for it:\n\n- all numeric features were converted to elements of SparseVector as-is;\n- all categorical features were converted to elements of SparseVector by interpreting the hashes as 32 bit numbers.\n\n\n\n### Code\n[_(back to toc)_](#table-of-contents)\n\nAll work was performed in Jupyter notebooks in Python. Notebooks:\n\n- [experiment_local.ipynb](notebooks/experiment_local.ipynb) was used for preparing the data and training of the local models;\n- [experiment_spark_lr.ipynb](notebooks/experiment_spark_lr.ipynb) and [experiment_spark_rf.ipynb](notebooks/experiment_spark_rf.ipynb) for training Spark.ML LogisticRegression and RandomForestClassifier accordingly.\n\n\n\n## Results\n\n### Local training - Vowpal Wabbit & XGBoost\n[_(back to toc)_](#table-of-contents)\n\n![ROC AUC](images/roc_auc.local.png) ![Log loss](images/log_loss.local.png) ![Train time](images/train_time.local.png) ![Maximum memory](images/maximum_memory.local.png) ![CPU load](images/cpu_load.local.png)\n\nSome observations:\n\n- our main concern about an out-of-core training of XGBoost was that it would not produce the same quality as its in-memory variant due to the approximate splitting algorithm; however, in-memory XGBoost and out-of-core XGBoost turned out to provide about the same level of quality, but out-of-core variant is about an order of magnitude slower;\n- in-memory XGBoost is about an order of magnitude slower than Vowpal Wabbit on the same amount of train data;\n- Vowpal Wabbit was able to give about the same quality as XGBoost trained on an order of magnitude smaller sample.\n\n\n\n### Distributed training - Spark.ML\n[_(back to toc)_](#table-of-contents)\n\n![ROC AUC](images/roc_auc.cluster.png) ![Log loss](images/log_loss.cluster.png) ![Train time](images/train_time.cluster.png)\n\nWe made the following conclusions:\n\n- RandomForestClassifier is quite slow, and it is even slower when the data consists of large vectors;\n- LogisticRegression is hard to set up for smaller samples and for bigger samples at the same time - either it overfits on small data or it cannot extract patterns due to more aggressive hashing trick.\n\n\n\n### Distributed training - Time vs. Cores\n[_(back to toc)_](#table-of-contents)\n\nTo check how model training scales to multi-core setup we made a quick test where we increased the number of cores and measured training time for every step. To make it fast we used a 10⁷ sample of train data and checked training time for a number of cores from 5 to 50 in steps of 5. In order to eliminate the uncertainty brought forth by running the test in parallel with production tasks, for this test we created a standalone Spark cluster running on three machines with a total of ≈50 cores and ≈200 GiB of memory.\n\n![Time vs. cores](images/time_vs_cores.png)\n\nTraining time dropped quite fast when we went from 5 to 15 cores but slowed down afterwards and completely ceased to improve by the mark of 40 cores (even growing a little on transition from 40 to 45 cores). The main idea we have extracted from this figure is that one should not increase amount of resources beyond minimum required, so that work distribution and aggregation overhead would be cheaper than potential improvement of speed by parallelization.\n\n\n\n### Comparison of local vs. remote\n[_(back to toc)_](#table-of-contents)\n\n![ROC AUC](images/roc_auc.local_and_cluster.png) ![Log loss](images/log_loss.local_and_cluster.png) ![Train time](images/train_time.local_and_cluster.png)\n\nWe can see that:\n\n- on large datasets (100 million of lines and more) Spark.ML is faster than both Vowpal Wabbit and XGBoost; maybe it is possible to make it even faster by finding the best cluster setup for each size of training sample (we had not done this work);\n- however it is slow when working with large vectors - steps should be taken in order to find a balance between quality and speed;\n- for small tasks Spark introduces overhead that can be more expensive than it is possible to gain by computing the task in parallel (well, this is true for parallel computing in general).\n\n\n\n## Conclusion\n[_(back to toc)_](#table-of-contents)\n\nThe best quality measured by logarithmic loss (which is a metric of choice for CTR prediction) was achived using XGBoost - no matter in-memory or out-of-core, as they both seem to be equal in quality - on a sample smaller in size than other models required for the same level of quality.\nHowever XGBoost is very slow on big samples in out-of-core setup thus it was not rational to test it on a 300kk sample and above (training the in-memory setup on large samples was also not possible due to memory restrictions).\nThe highest ROC AUC was reached by Vowpal Wabbit on one-billion-line train sample, strangely decreasing in quality by three-billion-line sample.\nSpark.ML LogisticRegression is quite close in quality to Vowpal Wabbit, and maybe it can be made even better by increasing the feature space size (which is 100k hashes in current setup).\nSpark.ML LogisticRegression appeared to be considerably faster than VW on billion-line samples and maybe it can be made even faster by optimizing the allocated resources.\nSpark.ML RandomForestClassifier stopped increasing in quality quite early and it is also quite slow.\n\n\n\n## Resources\n[_(back to toc)_](#table-of-contents)\n\nResults in table format can be found [here](results). Scala scripts used for faster conversion and sampling can be found [here](scripts/conversion) - these scripts can be used from [spark-shell](http://spark.apache.org/docs/latest/quick-start.html#basics) using `:load` command. Scripts for running VW & XGBoost and plotting outside of the Jupyter notebooks can be found [here](scripts/running).\n"
  },
  {
    "path": "notebooks/experiment_local.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"# Criteo 1 TiB benchmark\\n\",\n    \"\\n\",\n    \"In this experiment we will evalutate a number of machine learning tools on a varying size of train data to determine how fast they learn, how much memory they consume etc.\\n\",\n    \"\\n\",\n    \"We will assess Vowpal Wabbit and XGBoost in local mode, and Spark.ML models in cluster mode.\\n\",\n    \"\\n\",\n    \"We will use terabyte click logs released by Criteo and sample needed amount of data from them.\\n\",\n    \"\\n\",\n    \"This instance of experiment notebook focuses on data preparation and training VW & XGBoost locally.\\n\",\n    \"\\n\",\n    \"Let's go!\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"# Table of contents\\n\",\n    \"\\n\",\n    \"* [Configuration](#Configuration)\\n\",\n    \"* [Data preparation](#Data-preparation)\\n\",\n    \"  * [Criteo → LibSVM](#Criteo-→-LibSVM)\\n\",\n    \"  * [LibSVM → Train and test (sampling)](#LibSVM-→-Train-and-test-(sampling%29)\\n\",\n    \"  * [LibSVM train and test → VW train and test](#LibSVM-train-and-test-→-VW-train-and-test)\\n\",\n    \"  * [Local data](#Local-data)\\n\",\n    \"* [Local training](#Local-training)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"%load_ext autotime\\n\",\n    \"%matplotlib inline\\n\",\n    \"\\n\",\n    \"from __future__ import print_function\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"## Configuration\\n\",\n    \"[_(back to toc)_](#Table-of-contents)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Paths:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"criteo_data_remote_path = 'criteo/plain'\\n\",\n    \"libsvm_data_remote_path = 'criteo/libsvm'\\n\",\n    \"vw_data_remote_path = 'criteo/vw'\\n\",\n    \"\\n\",\n    \"local_data_path = 'criteo/data'\\n\",\n    \"local_results_path = 'criteo/results'\\n\",\n    \"local_runtime_path = 'criteo/runtime'\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"import os\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"criteo_day_template = os.path.join(criteo_data_remote_path, 'day_{}')\\n\",\n    \"libsvm_day_template = os.path.join(libsvm_data_remote_path, 'day_{}')\\n\",\n    \"vw_day_template = os.path.join(vw_data_remote_path, 'day_{}')\\n\",\n    \"\\n\",\n    \"libsvm_train_template = os.path.join(libsvm_data_remote_path, 'train', '{}')\\n\",\n    \"libsvm_test_template = os.path.join(libsvm_data_remote_path, 'test', '{}')\\n\",\n    \"vw_train_template = os.path.join(vw_data_remote_path, 'train', '{}')\\n\",\n    \"vw_test_template = os.path.join(vw_data_remote_path, 'test', '{}')\\n\",\n    \"\\n\",\n    \"local_libsvm_test_template = os.path.join(local_data_path, 'data.test.{}.libsvm')\\n\",\n    \"local_libsvm_train_template = os.path.join(local_data_path, 'data.train.{}.libsvm')\\n\",\n    \"local_vw_test_template = os.path.join(local_data_path, 'data.test.{}.vw')\\n\",\n    \"local_vw_train_template = os.path.join(local_data_path, 'data.train.{}.vw')\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"def ensure_directory_exists(path):\\n\",\n    \"    if not os.path.exists(path):\\n\",\n    \"        os.makedirs(path)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Days to work on:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"days = list(range(0, 23 + 1))\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Samples to take:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"train_samples = [\\n\",\n    \"    10000, 30000,  # tens of thousands\\n\",\n    \"    100000, 300000,  # hundreds of thousands\\n\",\n    \"    1000000, 3000000,  # millions\\n\",\n    \"    10000000, 30000000,  # tens of millions\\n\",\n    \"    100000000, 300000000,  # hundreds of millions\\n\",\n    \"    1000000000, 3000000000,  # billions\\n\",\n    \"]\\n\",\n    \"test_samples = [1000000]\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Spark configuration and initialization:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"total_cores = 256\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"executor_cores = 4\\n\",\n    \"executor_instances = total_cores / executor_cores\\n\",\n    \"memory_per_core = 4\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"app_name = 'Criteo experiment'\\n\",\n    \"\\n\",\n    \"master = 'yarn'\\n\",\n    \"\\n\",\n    \"settings = {\\n\",\n    \"    'spark.network.timeout': '600',\\n\",\n    \"    \\n\",\n    \"    'spark.driver.cores': '16',\\n\",\n    \"    'spark.driver.maxResultSize': '16G',\\n\",\n    \"    'spark.driver.memory': '32G',\\n\",\n    \"    \\n\",\n    \"    'spark.executor.cores': str(executor_cores),\\n\",\n    \"    'spark.executor.instances': str(executor_instances),\\n\",\n    \"    'spark.executor.memory': str(memory_per_core * executor_cores) + 'G',\\n\",\n    \"    \\n\",\n    \"    'spark.speculation': 'true',\\n\",\n    \"    'spark.yarn.queue': 'root.HungerGames',\\n\",\n    \"}\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"scrolled\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from pyspark.sql import SparkSession\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"builder = SparkSession.builder\\n\",\n    \"\\n\",\n    \"builder.appName(app_name)\\n\",\n    \"builder.master(master)\\n\",\n    \"for k, v in settings.items():\\n\",\n    \"    builder.config(k, v)\\n\",\n    \"\\n\",\n    \"spark = builder.getOrCreate()\\n\",\n    \"sc = spark.sparkContext\\n\",\n    \"\\n\",\n    \"sc.setLogLevel('ERROR')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Logging:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"import sys\\n\",\n    \"import logging\\n\",\n    \"reload(logging)\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"handler = logging.StreamHandler(stream=sys.stdout)\\n\",\n    \"formatter = logging.Formatter('[%(asctime)s] %(message)s')\\n\",\n    \"handler.setFormatter(formatter)\\n\",\n    \"\\n\",\n    \"ensure_directory_exists(local_runtime_path)\\n\",\n    \"file_handler = logging.FileHandler(filename=os.path.join(local_runtime_path, 'mylog.log'), mode='a')\\n\",\n    \"file_handler.setFormatter(formatter)\\n\",\n    \"\\n\",\n    \"logger = logging.getLogger()\\n\",\n    \"logger.addHandler(handler)\\n\",\n    \"logger.addHandler(file_handler)\\n\",\n    \"logger.setLevel(logging.INFO)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"logger.info('Spark version: %s.', spark.version)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"## Data preparation\\n\",\n    \"[_(back to toc)_](#Table-of-contents)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Poor man's HDFS API:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"def hdfs_exists(path):\\n\",\n    \"    l = !hadoop fs -ls $path 2>/dev/null\\n\",\n    \"    return len(l) != 0\\n\",\n    \"\\n\",\n    \"def hdfs_success(path):\\n\",\n    \"    return hdfs_exists(os.path.join(path, '_SUCCESS'))\\n\",\n    \"\\n\",\n    \"def hdfs_delete(path, recurse=False):\\n\",\n    \"    if recurse:\\n\",\n    \"        _ = !hadoop fs -rm -r $path\\n\",\n    \"    else:\\n\",\n    \"        _ = !hadoop fs -rm $path\\n\",\n    \"\\n\",\n    \"def hdfs_get(remote_path, local_path):\\n\",\n    \"    remote_path_glob = os.path.join(remote_path, 'part-*')\\n\",\n    \"    _ = !hadoop fs -cat $remote_path_glob >$local_path\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Load RDDs from one place and save them to another converted:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"def convert_chunked_data(input_path_template, output_path_template, chunks, load_rdd, convert_row, transform_rdd=None):\\n\",\n    \"    for chunk in chunks:\\n\",\n    \"        input_path = input_path_template.format(chunk)\\n\",\n    \"        output_path = output_path_template.format(chunk)\\n\",\n    \"\\n\",\n    \"        if hdfs_success(output_path):\\n\",\n    \"            logger.info('Chunk \\\"%s\\\" is already converted and saved to \\\"%s\\\", skipping.', chunk, output_path)\\n\",\n    \"            continue\\n\",\n    \"\\n\",\n    \"        logger.info('Reading chunk \\\"%s\\\" data from \\\"%s\\\".', chunk, input_path)\\n\",\n    \"        rdd = load_rdd(input_path)\\n\",\n    \"\\n\",\n    \"        if hdfs_exists(output_path):\\n\",\n    \"            logger.info('Cleaning \\\"%s\\\".', output_path)\\n\",\n    \"            hdfs_delete(output_path, recurse=True)\\n\",\n    \"\\n\",\n    \"        logger.info('Processing and saving to \\\"%s\\\".', output_path)\\n\",\n    \"        rdd = rdd.map(convert_row)\\n\",\n    \"        \\n\",\n    \"        if transform_rdd is not None:\\n\",\n    \"            rdd = transform_rdd(rdd)\\n\",\n    \"        \\n\",\n    \"        rdd.saveAsTextFile(output_path)\\n\",\n    \"\\n\",\n    \"        logger.info('Done with chunk \\\"%s\\\".', chunk)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"### Criteo → LibSVM\\n\",\n    \"[_(back to toc)_](#Table-of-contents)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Criteo RDD is actually a DataFrame:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"def load_criteo_rdd(path):\\n\",\n    \"    return (\\n\",\n    \"        spark\\n\",\n    \"        .read\\n\",\n    \"        .option('header', 'false')\\n\",\n    \"        .option('inferSchema', 'true')\\n\",\n    \"        .option('delimiter', '\\\\t')\\n\",\n    \"        .csv(path)\\n\",\n    \"        .rdd\\n\",\n    \"    )\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Simply add an index to each existing column except the first one which is a target:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"def criteo_to_libsvm(row):\\n\",\n    \"    return (\\n\",\n    \"        str(row[0])\\n\",\n    \"        + ' '\\n\",\n    \"        + ' '.join(\\n\",\n    \"            [\\n\",\n    \"                # integer features\\n\",\n    \"                str(i) + ':' + str(row[i])\\n\",\n    \"                for i in range(1, 13 + 1)\\n\",\n    \"                if row[i] is not None\\n\",\n    \"            ] + [\\n\",\n    \"                # string features converted from hex to int\\n\",\n    \"                str(i) + ':' + str(int(row[i], 16))\\n\",\n    \"                for i in range(14, 39 + 1)\\n\",\n    \"                if row[i] is not None\\n\",\n    \"            ]\\n\",\n    \"        )\\n\",\n    \"    )\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Do it for all days:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"convert_chunked_data(criteo_day_template, libsvm_day_template, days, load_criteo_rdd, criteo_to_libsvm)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"### LibSVM → Train and test (sampling)\\n\",\n    \"[_(back to toc)_](#Table-of-contents)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Let's name samples as their shortened \\\"engineering\\\" notation - e.g. 1e5 is 100k etc.:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"def sample_name(sample):\\n\",\n    \"    return str(sample)[::-1].replace('000', 'k')[::-1]\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Load data, sample a bit more than needed and cut at exact desired number of lines by zipping with index and filtering upto required index:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"oversample = 1.03\\n\",\n    \"sampled_partitions = 256\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"def sample_and_save(input_path_template, output_path_template, days, samples):\\n\",\n    \"    union = None\\n\",\n    \"    union_count = None\\n\",\n    \"    \\n\",\n    \"    for sample in samples:\\n\",\n    \"        name = sample_name(sample)\\n\",\n    \"        output_path = output_path_template.format(name)\\n\",\n    \"        \\n\",\n    \"        if hdfs_success(output_path):\\n\",\n    \"            logger.info('Sample \\\"%s\\\" is already written to \\\"%s\\\", skipping.', sample, output_path)\\n\",\n    \"            continue\\n\",\n    \"            \\n\",\n    \"        logger.info('Preparing to write sample to \\\"%s\\\".', output_path)\\n\",\n    \"        \\n\",\n    \"        if union is None:\\n\",\n    \"            rdds = map(lambda day: sc.textFile(input_path_template.format(day)), days)\\n\",\n    \"            union = reduce(lambda left, right: left.union(right), rdds)\\n\",\n    \"\\n\",\n    \"            union_count = union.count()\\n\",\n    \"            logger.info('Total number of lines for days \\\"%s\\\" is \\\"%s\\\".', days, union_count)\\n\",\n    \"            \\n\",\n    \"        ratio = float(sample) / union_count\\n\",\n    \"        \\n\",\n    \"        sampled_union = (\\n\",\n    \"            union\\n\",\n    \"            .sample(False, min(1.0, oversample * ratio))\\n\",\n    \"            .zipWithIndex()\\n\",\n    \"            .filter(lambda z: z[1] < sample)\\n\",\n    \"            .map(lambda z: z[0])\\n\",\n    \"        )\\n\",\n    \"        \\n\",\n    \"        if hdfs_exists(output_path):\\n\",\n    \"            logger.info('Cleaning \\\"%s\\\".', output_path)\\n\",\n    \"            hdfs_delete(output_path, recurse=True)\\n\",\n    \"            \\n\",\n    \"        logger.info('Writing sample \\\"%s\\\" to \\\"%s\\\".', sample, output_path)\\n\",\n    \"        sampled_union.coalesce(sampled_partitions).saveAsTextFile(output_path)\\n\",\n    \"        \\n\",\n    \"        logger.info('Saved \\\"%s\\\" lines to \\\"%s\\\".', sc.textFile(output_path).count(), output_path)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Sample all LibSVM data:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"sample_and_save(libsvm_day_template, libsvm_test_template, days[-1:], test_samples)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"sample_and_save(libsvm_day_template, libsvm_train_template, days[:-1], train_samples)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"### LibSVM train and test → VW train and test\\n\",\n    \"[_(back to toc)_](#Table-of-contents)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"LibSVM RDD is a text file:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"def load_libsvm_rdd(path):\\n\",\n    \"    return sc.textFile(path)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Conversion is trivial - we only have to map target to {-1, 1} and convert categorical features to VW feature names as a whole:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"def libsvm_to_vw(line):\\n\",\n    \"    parts = line.split(' ')\\n\",\n    \"    parts[0] = '1 |' if parts[0] == '1' else '-1 |'\\n\",\n    \"    for i in range(1, len(parts)):\\n\",\n    \"        index, _, value = parts[i].partition(':')\\n\",\n    \"        if int(index) >= 14:\\n\",\n    \"            parts[i] = index + '_' + value\\n\",\n    \"    return ' '.join(parts)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Also, data for VW should be well shuffled:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"import hashlib\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"def calculate_hash(something):\\n\",\n    \"    m = hashlib.md5()\\n\",\n    \"    m.update(str(something))\\n\",\n    \"    return m.hexdigest()\\n\",\n    \"\\n\",\n    \"def random_sort(rdd):\\n\",\n    \"    return (\\n\",\n    \"        rdd\\n\",\n    \"        .zipWithIndex()\\n\",\n    \"        .sortBy(lambda z: calculate_hash(z[1]))\\n\",\n    \"        .map(lambda z: z[0])\\n\",\n    \"    )\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Convert all LibSVM samples:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"convert_chunked_data(libsvm_test_template, vw_test_template, [sample_name(sample) for sample in test_samples], load_libsvm_rdd, libsvm_to_vw, transform_rdd=random_sort)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"scrolled\": false\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"convert_chunked_data(libsvm_train_template, vw_train_template, [sample_name(sample) for sample in train_samples], load_libsvm_rdd, libsvm_to_vw, transform_rdd=random_sort)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Spark is no longer needed:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"spark.stop()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"### Local data\\n\",\n    \"[_(back to toc)_](#Table-of-contents)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Download all sampled data to local directory:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"ensure_directory_exists(local_data_path)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"def count_lines(path):\\n\",\n    \"    with open(path) as f:\\n\",\n    \"        for i, _ in enumerate(f):\\n\",\n    \"            pass\\n\",\n    \"    return i + 1\\n\",\n    \"\\n\",\n    \"def download_data(remote_template, local_template, samples):\\n\",\n    \"    for sample in samples:\\n\",\n    \"        name = sample_name(sample)\\n\",\n    \"        remote_path = remote_template.format(name)\\n\",\n    \"        local_path = local_template.format(name)\\n\",\n    \"        if os.path.exists(local_path):\\n\",\n    \"            count = count_lines(local_path)\\n\",\n    \"            if count == sample:\\n\",\n    \"                logger.info('File \\\"%s\\\" is already loaded, skipping.', local_path)\\n\",\n    \"                continue\\n\",\n    \"            else:\\n\",\n    \"                logger.info('File \\\"%s\\\" already exists but number of lines \\\"%s\\\" is wrong (must be \\\"%s\\\"), reloading.', local_path, count, sample)\\n\",\n    \"        logger.info('Loading file \\\"%s\\\" as local file \\\"%s\\\".', remote_path, local_path)\\n\",\n    \"        hdfs_get(remote_path, local_path)\\n\",\n    \"        count = count_lines(local_path)\\n\",\n    \"        logger.info('File loaded to \\\"%s\\\", number of lines is \\\"%s\\\".', local_path, count)\\n\",\n    \"        assert count == sample, 'File \\\"{}\\\" contains wrong number of lines \\\"{}\\\" (must be \\\"{}\\\").'.format(local_path, count, sample)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"download_data(libsvm_test_template, local_libsvm_test_template, test_samples)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"download_data(libsvm_train_template, local_libsvm_train_template, train_samples)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"download_data(vw_test_template, local_vw_test_template, test_samples)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"download_data(vw_train_template, local_vw_train_template, train_samples)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"## Local training\\n\",\n    \"[_(back to toc)_](#Table-of-contents)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Measuring model quality and ML engine technical metrics:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"import sys \\n\",\n    \"from matplotlib import pyplot\\n\",\n    \"from sklearn.metrics import (\\n\",\n    \"    auc,\\n\",\n    \"    log_loss,\\n\",\n    \"    roc_curve,\\n\",\n    \")\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"def measure(engine, sample, test_file, time_file, predictions_file):\\n\",\n    \"    \\n\",\n    \"    def get_last_in_line(s):\\n\",\n    \"        return s.rstrip().split( )[-1]\\n\",\n    \"\\n\",\n    \"    def parse_elapsed_time(s):\\n\",\n    \"        return reduce(lambda a, b: a * 60 + b, map(float, get_last_in_line(s).split(':')))\\n\",\n    \"\\n\",\n    \"    def parse_max_memory(s):\\n\",\n    \"        return int(get_last_in_line(s)) * 1024\\n\",\n    \"\\n\",\n    \"    def parse_cpu(s):\\n\",\n    \"        return float(get_last_in_line(s).rstrip('%')) / 100 \\n\",\n    \"\\n\",\n    \"\\n\",\n    \"    elapsed = -1\\n\",\n    \"    memory = -1\\n\",\n    \"    cpu = -1\\n\",\n    \"\\n\",\n    \"    with open(time_file, 'rb') as f:\\n\",\n    \"        for line in f:\\n\",\n    \"            if 'Elapsed (wall clock) time' in line:\\n\",\n    \"                elapsed = parse_elapsed_time(line)\\n\",\n    \"            elif 'Maximum resident set size' in line:\\n\",\n    \"                memory = parse_max_memory(line)\\n\",\n    \"            elif 'Percent of CPU' in line:\\n\",\n    \"                cpu = parse_cpu(line)\\n\",\n    \"\\n\",\n    \"    with open(test_file, 'rb') as f:\\n\",\n    \"        labels = [line.rstrip().split(' ')[0] == '1' for line in f]\\n\",\n    \"\\n\",\n    \"    with open(predictions_file, 'rb') as f:\\n\",\n    \"        scores = [float(line.rstrip().split(' ')[0]) for line in f]\\n\",\n    \"\\n\",\n    \"    fpr, tpr, _ = roc_curve(labels, scores)\\n\",\n    \"    roc_auc = auc(fpr, tpr)\\n\",\n    \"    ll = log_loss(labels, scores)\\n\",\n    \"    \\n\",\n    \"    figure = pyplot.figure(figsize=(6, 6))\\n\",\n    \"    pyplot.plot(fpr, tpr, linewidth=2.0)\\n\",\n    \"    pyplot.plot([0, 1], [0, 1], 'k--')\\n\",\n    \"    pyplot.xlabel('FPR')\\n\",\n    \"    pyplot.ylabel('TPR')\\n\",\n    \"    pyplot.title('{} {} - {:.3f} ROC AUC'.format(engine, sample_name(sample), roc_auc))\\n\",\n    \"    pyplot.show()\\n\",\n    \"\\n\",\n    \"    return {\\n\",\n    \"        'Engine': engine,\\n\",\n    \"        'Train size': sample,\\n\",\n    \"        'ROC AUC': roc_auc,\\n\",\n    \"        'Log loss': ll,\\n\",\n    \"        'Train time': elapsed,\\n\",\n    \"        'Maximum memory': memory,\\n\",\n    \"        'CPU load': cpu,\\n\",\n    \"    }\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Settings for VW & XGBoost and how to run them; I use (a little bit patched for correctness sake) GNU Time to measure running time, CPU load and memory consumption; configurations for VW & XGBoost are obtained via Hyperopt:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"def get_time_command_and_file(train_file):\\n\",\n    \"    time_file = train_file + '.time'\\n\",\n    \"    return [\\n\",\n    \"        '/usr/local/bin/time',\\n\",\n    \"        '-v',\\n\",\n    \"        '--output=' + time_file,\\n\",\n    \"    ], time_file\\n\",\n    \"\\n\",\n    \"def get_vw_commands_and_predictions_file(train_file, test_file):\\n\",\n    \"    model_file = train_file + '.model'\\n\",\n    \"    predictions_file = test_file + '.predictions'\\n\",\n    \"    return [\\n\",\n    \"        'vw83',\\n\",\n    \"        '--link=logistic',\\n\",\n    \"        '--loss_function=logistic',\\n\",\n    \"        '-b', '29',\\n\",\n    \"        '-l', '0.3',\\n\",\n    \"        '--initial_t', '1',\\n\",\n    \"        '--decay_learning_rate', '0.5',\\n\",\n    \"        '--power_t', '0.5',\\n\",\n    \"        '--l1', '1e-15',\\n\",\n    \"        '--l2', '0',\\n\",\n    \"        '-d', train_file,\\n\",\n    \"        '-f', model_file,\\n\",\n    \"    ], [\\n\",\n    \"        'vw83',\\n\",\n    \"        '--loss_function=logistic',\\n\",\n    \"        '-t',\\n\",\n    \"        '-i', model_file,\\n\",\n    \"        '-d', test_file,\\n\",\n    \"        '-p', predictions_file,\\n\",\n    \"    ], predictions_file\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"xgboost_conf = [\\n\",\n    \"    'booster = gbtree',\\n\",\n    \"    'objective = binary:logistic',\\n\",\n    \"    'nthread = 24',\\n\",\n    \"    'eval_metric = logloss',\\n\",\n    \"    'max_depth = 7',\\n\",\n    \"    'num_round = 200',\\n\",\n    \"    'eta = 0.2',\\n\",\n    \"    'gamma = 0.4',\\n\",\n    \"    'subsample = 0.8',\\n\",\n    \"    'colsample_bytree = 0.8',\\n\",\n    \"    'min_child_weight = 20',\\n\",\n    \"    'alpha = 3',\\n\",\n    \"    'lambda = 100',\\n\",\n    \"]\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"def get_xgboost_commands_and_predictions_file(train_file, test_file, cache=False):\\n\",\n    \"    config_file = os.path.join(local_runtime_path, 'xgb.conf')\\n\",\n    \"    ensure_directory_exists(local_runtime_path)\\n\",\n    \"    with open(config_file, 'wb') as f:\\n\",\n    \"        for line in xgboost_conf:\\n\",\n    \"            print(line, file=f)\\n\",\n    \"    model_file = train_file + '.model'\\n\",\n    \"    predictions_file = test_file + '.predictions'\\n\",\n    \"    if cache:\\n\",\n    \"        train_file = train_file + '#' + train_file + '.cache'\\n\",\n    \"    return [\\n\",\n    \"        'xgboost',\\n\",\n    \"        config_file,\\n\",\n    \"        'data=' + train_file,\\n\",\n    \"        'model_out=' + model_file,\\n\",\n    \"    ], [\\n\",\n    \"        'xgboost',\\n\",\n    \"        config_file,\\n\",\n    \"        'task=pred',\\n\",\n    \"        'test:data=' + test_file,\\n\",\n    \"        'model_in=' + model_file,\\n\",\n    \"        'name_pred=' + predictions_file,\\n\",\n    \"    ], predictions_file\\n\",\n    \"\\n\",\n    \"def get_xgboost_ooc_commands_and_predictions_file(train_file, test_file):\\n\",\n    \"    return get_xgboost_commands_and_predictions_file(train_file, test_file, cache=True)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"engines = {\\n\",\n    \"    'vw': (get_vw_commands_and_predictions_file, local_vw_train_template, local_vw_test_template),\\n\",\n    \"    'xgb': (get_xgboost_commands_and_predictions_file, local_libsvm_train_template, local_libsvm_test_template),\\n\",\n    \"    'xgb.ooc': (get_xgboost_ooc_commands_and_predictions_file, local_libsvm_train_template, local_libsvm_test_template),\\n\",\n    \"}\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Train & test everything:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"scrolled\": false\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"import subprocess\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"measurements = []\\n\",\n    \"\\n\",\n    \"for sample in train_samples:\\n\",\n    \"    for engine in engines:\\n\",\n    \"        logger.info('Training \\\"%s\\\" on \\\"%s\\\" lines of data.', engine, sample)\\n\",\n    \"        \\n\",\n    \"        get_commands_and_predictions_file, train_template, test_template = engines[engine]\\n\",\n    \"\\n\",\n    \"        train_file = train_template.format(sample_name(sample))\\n\",\n    \"        test_file = test_template.format(sample_name(test_samples[0]))\\n\",\n    \"        logger.info('Will train on \\\"%s\\\" and test on \\\"%s\\\".', train_file, test_file)\\n\",\n    \"\\n\",\n    \"        command_time, time_file = get_time_command_and_file(train_file)\\n\",\n    \"        command_engine_train, command_engine_test, predictions_file = get_commands_and_predictions_file(train_file, test_file)\\n\",\n    \"\\n\",\n    \"        logger.info('Performing train.')\\n\",\n    \"        subprocess.call(command_time + command_engine_train)\\n\",\n    \"\\n\",\n    \"        logger.info('Performing test.')\\n\",\n    \"        subprocess.call(command_engine_test)\\n\",\n    \"\\n\",\n    \"        logger.info('Measuring results.')\\n\",\n    \"        measurement = measure(engine, sample, test_file, time_file, predictions_file)\\n\",\n    \"        logger.info(measurement)\\n\",\n    \"        measurements.append(measurement)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Load measurements:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"import pandas\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"measurements_df = pandas.DataFrame(measurements).sort_values(by=['Engine', 'Train size'])\\n\",\n    \"measurements_df\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Plot measurements:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"scrolled\": false\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"def extract_data_for_plotting(df, what):\\n\",\n    \"    return reduce(\\n\",\n    \"        lambda left, right: pandas.merge(left, right, how='outer', on='Train size'),\\n\",\n    \"        map(\\n\",\n    \"            lambda name: df[df.Engine == name][['Train size', what]].rename(columns={what: name}),\\n\",\n    \"            df.Engine.unique(),\\n\",\n    \"        ),\\n\",\n    \"    )   \\n\",\n    \"\\n\",\n    \"def plot_stuff(df, what, ylabel=None, **kwargs):\\n\",\n    \"    data = extract_data_for_plotting(df, what).set_index('Train size')\\n\",\n    \"    ax = data.plot(marker='o', figsize=(6, 6), title=what, grid=True, linewidth=2.0, **kwargs)  # xlim=(1e4, 4e9)\\n\",\n    \"    if ylabel is not None:\\n\",\n    \"        ax.set_ylabel(ylabel)\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"plot_stuff(measurements_df, 'ROC AUC', logx=True)\\n\",\n    \"plot_stuff(measurements_df, 'Log loss', logx=True)\\n\",\n    \"plot_stuff(measurements_df, 'Train time', loglog=True, ylabel='s')\\n\",\n    \"plot_stuff(measurements_df, 'Maximum memory', loglog=True, ylabel='bytes')\\n\",\n    \"plot_stuff(measurements_df, 'CPU load', logx=True)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": []\n  }\n ],\n \"metadata\": {\n  \"kernelspec\": {\n   \"display_name\": \"Python 2\",\n   \"language\": \"python\",\n   \"name\": \"python2\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 2\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython2\",\n   \"version\": \"2.7.9\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n"
  },
  {
    "path": "notebooks/experiment_spark_lr.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"source\": [\n    \"# Criteo 1 TiB benchmark - Spark.ML logistic regression\\n\",\n    \"\\n\",\n    \"Specialization of the experiment notebook for Spark.ML logistic regression.\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"# Table of contents\\n\",\n    \"\\n\",\n    \"* [Configuration](#Configuration)\\n\",\n    \"* [Distributed training](#Distributed-training)\\n\",\n    \"* [End](#End)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"%load_ext autotime\\n\",\n    \"%matplotlib inline\\n\",\n    \"\\n\",\n    \"from __future__ import print_function\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"## Configuration\\n\",\n    \"[_(back to toc)_](#Table-of-contents)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Paths:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"libsvm_data_remote_path = 'criteo/libsvm'\\n\",\n    \"local_runtime_path = 'criteo/runtime'\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"import os\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"libsvm_train_template = os.path.join(libsvm_data_remote_path, 'train', '{}')\\n\",\n    \"libsvm_test_template = os.path.join(libsvm_data_remote_path, 'test', '{}')\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"def ensure_directory_exists(path):\\n\",\n    \"    if not os.path.exists(path):\\n\",\n    \"        os.makedirs(path)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Samples to take:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"train_samples = [\\n\",\n    \"    10000, 30000,  # tens of thousands\\n\",\n    \"    100000, 300000,  # hundreds of thousands\\n\",\n    \"    1000000, 3000000,  # millions\\n\",\n    \"    10000000, 30000000,  # tens of millions\\n\",\n    \"    100000000, 300000000,  # hundreds of millions\\n\",\n    \"    1000000000, 3000000000,  # billions\\n\",\n    \"]\\n\",\n    \"\\n\",\n    \"test_samples = [1000000]\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Spark configuration and initialization:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"total_cores = 256\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"executor_cores = 4\\n\",\n    \"executor_instances = total_cores / executor_cores\\n\",\n    \"memory_per_core = 2\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"app_name = 'Criteo experiment - LR on 128 cores'\\n\",\n    \"\\n\",\n    \"master = 'yarn'\\n\",\n    \"\\n\",\n    \"settings = {\\n\",\n    \"    'spark.network.timeout': '600',\\n\",\n    \"    \\n\",\n    \"    'spark.driver.cores': '16',\\n\",\n    \"    'spark.driver.maxResultSize': '16G',\\n\",\n    \"    'spark.driver.memory': '32G',\\n\",\n    \"    \\n\",\n    \"    'spark.executor.cores': str(executor_cores),\\n\",\n    \"    'spark.executor.instances': str(executor_instances),\\n\",\n    \"    'spark.executor.memory': str(memory_per_core * executor_cores) + 'G',\\n\",\n    \"    \\n\",\n    \"    'spark.speculation': 'true',\\n\",\n    \"    \\n\",\n    \"    'spark.yarn.queue': 'root.HungerGames',\\n\",\n    \"}\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true,\n    \"scrolled\": false\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from pyspark.sql import SparkSession\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"builder = SparkSession.builder\\n\",\n    \"\\n\",\n    \"builder.appName(app_name)\\n\",\n    \"builder.master(master)\\n\",\n    \"for k, v in settings.items():\\n\",\n    \"    builder.config(k, v)\\n\",\n    \"\\n\",\n    \"spark = builder.getOrCreate()\\n\",\n    \"sc = spark.sparkContext\\n\",\n    \"\\n\",\n    \"sc.setLogLevel('ERROR')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Logging:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"import sys\\n\",\n    \"import logging\\n\",\n    \"reload(logging)\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"handler = logging.StreamHandler(stream=sys.stdout)\\n\",\n    \"formatter = logging.Formatter('[%(asctime)s] %(message)s')\\n\",\n    \"handler.setFormatter(formatter)\\n\",\n    \"\\n\",\n    \"ensure_directory_exists(local_runtime_path)\\n\",\n    \"file_handler = logging.FileHandler(filename=os.path.join(local_runtime_path, 'mylog.log'), mode='a')\\n\",\n    \"file_handler.setFormatter(formatter)\\n\",\n    \"\\n\",\n    \"logger = logging.getLogger()\\n\",\n    \"logger.addHandler(handler)\\n\",\n    \"logger.addHandler(file_handler)\\n\",\n    \"logger.setLevel(logging.DEBUG)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"logger.info('Spark version: %s.', spark.version)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Plot measurements:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true,\n    \"scrolled\": false\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"import pandas\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"def extract_data_for_plotting(df, what):\\n\",\n    \"    return reduce(\\n\",\n    \"        lambda left, right: pandas.merge(left, right, how='outer', on='Train size'),\\n\",\n    \"        map(\\n\",\n    \"            lambda name: df[df.Engine == name][['Train size', what]].rename(columns={what: name}),\\n\",\n    \"            df.Engine.unique(),\\n\",\n    \"        ),\\n\",\n    \"    )   \\n\",\n    \"\\n\",\n    \"def plot_stuff(df, what, ylabel=None, **kwargs):\\n\",\n    \"    data = extract_data_for_plotting(df, what).set_index('Train size')\\n\",\n    \"    ax = data.plot(marker='o', figsize=(6, 6), title=what, grid=True, linewidth=2.0, **kwargs)  # xlim=(1e4, 4e9)\\n\",\n    \"    if ylabel is not None:\\n\",\n    \"        ax.set_ylabel(ylabel)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Let's name samples as their shortened \\\"engineering\\\" notation - 1e5 is 100k etc.:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"def sample_name(sample):\\n\",\n    \"    return str(sample)[::-1].replace('000', 'k')[::-1]\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"## Distributed training\\n\",\n    \"[_(back to toc)_](#Table-of-contents)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Loading of LibSVM data as Spark.ML dataset:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"import hashlib\\n\",\n    \"import math\\n\",\n    \"import struct\\n\",\n    \"from pyspark.ml.linalg import SparseVector\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"total_features = 100000\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"def hash_fun(x):\\n\",\n    \"    return int(struct.unpack('L', hashlib.md5(x).digest()[:8])[0] % total_features)\\n\",\n    \"\\n\",\n    \"def parse_kv_pair(kv):\\n\",\n    \"    k, _, v = kv.partition(':')\\n\",\n    \"    return int(k), int(v)\\n\",\n    \"\\n\",\n    \"def parse_libsvm_line(line):\\n\",\n    \"    parts = line.split(' ')\\n\",\n    \"    \\n\",\n    \"    label = int(parts[0])\\n\",\n    \"    pairs = map(parse_kv_pair, parts[1:])\\n\",\n    \"    \\n\",\n    \"    for i in range(len(pairs)):\\n\",\n    \"        k, v = pairs[i]\\n\",\n    \"        if k < 14:\\n\",\n    \"            if v > 2:\\n\",\n    \"                pairs[i] = (k, int(math.log(v) ** 2))\\n\",\n    \"        else:\\n\",\n    \"            pairs[i] = (k, '{:08x}'.format(v))\\n\",\n    \"    \\n\",\n    \"    indices = sorted({hash_fun('{}_{}'.format(k, v)) for k, v in pairs})\\n\",\n    \"    values = [1.0] * len(indices)\\n\",\n    \"    \\n\",\n    \"    return (label, SparseVector(total_features, indices, values))\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"task_splitting = 1  # tasks per core\\n\",\n    \"\\n\",\n    \"def load_ml_data(template, sample):\\n\",\n    \"    path = template.format(sample_name(sample))\\n\",\n    \"    return sc.textFile(path).map(parse_libsvm_line).toDF(['label', 'features']).repartition(executor_cores * executor_instances * task_splitting)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Evaluating a model:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from matplotlib import pyplot\\n\",\n    \"from sklearn.metrics import auc, log_loss, roc_curve\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"def calculate_roc(predictions):\\n\",\n    \"    labels, scores = zip(*predictions.rdd.map(lambda row: (row.label, row.probability[1])).collect())\\n\",\n    \"    fpr, tpr, _ = roc_curve(labels, scores)\\n\",\n    \"    roc_auc = auc(fpr, tpr)\\n\",\n    \"    ll = log_loss(labels, scores)\\n\",\n    \"    return fpr, tpr, roc_auc, ll\\n\",\n    \"\\n\",\n    \"def evaluate_model(name, model, test, train=None):\\n\",\n    \"    metrics = dict()\\n\",\n    \"    \\n\",\n    \"    figure = pyplot.figure(figsize=(6, 6))\\n\",\n    \"    ax = figure.gca()\\n\",\n    \"    ax.set_title('ROC - ' + name)\\n\",\n    \"    \\n\",\n    \"    if train is not None:\\n\",\n    \"        train_predictions = model.transform(train)\\n\",\n    \"        train_fpr, train_tpr, train_roc_auc, train_log_loss = calculate_roc(train_predictions)\\n\",\n    \"        \\n\",\n    \"        metrics['train_roc_auc'] = train_roc_auc\\n\",\n    \"        metrics['train_log_loss'] = train_log_loss\\n\",\n    \"        \\n\",\n    \"        ax.plot(train_fpr, train_tpr, linewidth=2.0, label='train (auc = {:.3f})'.format(train_roc_auc))\\n\",\n    \"    \\n\",\n    \"    test_predictions = model.transform(test)\\n\",\n    \"    test_fpr, test_tpr, test_roc_auc, test_log_loss = calculate_roc(test_predictions)\\n\",\n    \"    \\n\",\n    \"    metrics['test_roc_auc'] = test_roc_auc\\n\",\n    \"    metrics['test_log_loss'] = test_log_loss\\n\",\n    \"    \\n\",\n    \"    ax.plot(test_fpr, test_tpr, linewidth=2.0, label='test (auc = {:.3f})'.format(test_roc_auc))\\n\",\n    \"    \\n\",\n    \"    ax.plot([0.0, 1.0], [0.0, 1.0], linestyle='--', c='gray')\\n\",\n    \"    ax.legend()\\n\",\n    \"    pyplot.show()\\n\",\n    \"    \\n\",\n    \"    return metrics\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Models to work on:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from pyspark.ml.classification import (\\n\",\n    \"    LogisticRegression,\\n\",\n    \")\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"classifiers = {\\n\",\n    \"    'lr': LogisticRegression(regParam=0.03),\\n\",\n    \"}\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Monkey-patch RDDs and DataFrames for context persistence:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"import pyspark\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"def enter_method(self):\\n\",\n    \"    self.persist()\\n\",\n    \"\\n\",\n    \"def exit_method(self,exc_type, exc, traceback):\\n\",\n    \"    self.unpersist()\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"pyspark.sql.dataframe.DataFrame.__enter__ = enter_method\\n\",\n    \"pyspark.sql.dataframe.DataFrame.__exit__ = exit_method\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Do distributed training:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true,\n    \"scrolled\": false\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"import time\\n\",\n    \"\\n\",\n    \"from pyspark.ml.evaluation import BinaryClassificationEvaluator\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"test_sample = test_samples[-1]\\n\",\n    \"\\n\",\n    \"evaluator = BinaryClassificationEvaluator(labelCol='label', rawPredictionCol='probability', metricName='areaUnderROC')\\n\",\n    \"\\n\",\n    \"new_quality_data = []\\n\",\n    \"new_data_rows = []\\n\",\n    \"\\n\",\n    \"test_dfs = dict()\\n\",\n    \"\\n\",\n    \"logger.info('Loading \\\"%s\\\" test samples.', test_sample)\\n\",\n    \"test_df = load_ml_data(libsvm_test_template, test_sample)\\n\",\n    \"with test_df:\\n\",\n    \"    logger.info('Loaded \\\"%s\\\" lines.', test_df.count())\\n\",\n    \"\\n\",\n    \"    for train_sample in train_samples:\\n\",\n    \"\\n\",\n    \"        logger.info('Working on \\\"%s\\\" train sample.', train_sample)\\n\",\n    \"\\n\",\n    \"        for classifier_name, classifier in classifiers.items():\\n\",\n    \"\\n\",\n    \"            logger.info('Training a model \\\"%s\\\" on sample \\\"%s\\\".', classifier_name, train_sample)\\n\",\n    \"\\n\",\n    \"            logger.info('Loading \\\"%s\\\" train samples.', train_sample)\\n\",\n    \"            train_df = load_ml_data(libsvm_train_template, train_sample)\\n\",\n    \"            with train_df:\\n\",\n    \"                logger.info('Loaded \\\"%s\\\" lines.', train_df.count())\\n\",\n    \"                \\n\",\n    \"                logger.info('Training a model \\\"%s\\\" on sample \\\"%s\\\".', classifier_name, train_sample)\\n\",\n    \"\\n\",\n    \"                start = time.time()\\n\",\n    \"                model = classifier.fit(train_df)\\n\",\n    \"                duration = time.time() - start\\n\",\n    \"\\n\",\n    \"                logger.info('Training a model \\\"%s\\\" on sample \\\"%s\\\" took \\\"%g\\\" seconds.', classifier_name, train_sample, duration)\\n\",\n    \"\\n\",\n    \"                logger.info('Evaluating the model \\\"%s\\\" trained on sample \\\"%s\\\".', classifier_name, train_sample)\\n\",\n    \"                metrics = evaluate_model(classifier_name + ' - ' + sample_name(train_sample), model, test_df, train=(train_df if train_sample <= 1000000 else None))\\n\",\n    \"\\n\",\n    \"                test_predictions = model.transform(test_df)\\n\",\n    \"                ml_metric_value = evaluator.evaluate(test_predictions)\\n\",\n    \"\\n\",\n    \"                logger.info(\\n\",\n    \"                    'For the model \\\"%s\\\" trained on sample \\\"%s\\\" metrics are: \\\"%s\\\"; ROC AUC calculated by Spark is \\\"%s\\\".',\\n\",\n    \"                    classifier_name,\\n\",\n    \"                    train_sample,\\n\",\n    \"                    metrics,\\n\",\n    \"                    ml_metric_value,\\n\",\n    \"                )\\n\",\n    \"\\n\",\n    \"                data_row = {\\n\",\n    \"                    'Train size': train_sample,\\n\",\n    \"                    'ROC AUC': metrics['test_roc_auc'],\\n\",\n    \"                    'Log loss': metrics['test_log_loss'],\\n\",\n    \"                    'Duration': duration,\\n\",\n    \"                    'Engine': classifier_name,\\n\",\n    \"                }\\n\",\n    \"                new_quality_data.append(data_row)\\n\",\n    \"                data_row_string = '\\\\t'.join(str(data_row[field]) for field in ['Engine', 'Train size', 'ROC AUC', 'Log loss', 'Duration'])\\n\",\n    \"                new_data_rows.append(data_row_string)\\n\",\n    \"                logger.info('Data row: \\\"%s\\\".', data_row_string)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true,\n    \"scrolled\": false\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"measurements_df = pandas.DataFrame(new_quality_data).sort_values(by=['Train size'])\\n\",\n    \"plot_stuff(measurements_df, 'ROC AUC', logx=True)\\n\",\n    \"plot_stuff(measurements_df, 'Log loss', logx=True, ylim=(0.13, 0.18))\\n\",\n    \"plot_stuff(measurements_df, 'Duration', loglog=True, ylabel='s')\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"for row in new_data_rows:\\n\",\n    \"    print(row)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"## End\\n\",\n    \"[_(back to toc)_](#Table-of-contents)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Work done, stop Spark:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"spark.stop()\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"kernelspec\": {\n   \"display_name\": \"Python 3\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.6.1\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n"
  },
  {
    "path": "notebooks/experiment_spark_rf.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"# Criteo 1 TiB benchmark - Spark.ML random forest\\n\",\n    \"\\n\",\n    \"Specialization of the experimental notebook for Spark.ML random forest.\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"# Table of contents\\n\",\n    \"\\n\",\n    \"* [Configuration](#Configuration)\\n\",\n    \"* [Distributed training](#Distributed-training)\\n\",\n    \"* [End](#End)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"%load_ext autotime\\n\",\n    \"%matplotlib inline\\n\",\n    \"\\n\",\n    \"from __future__ import print_function\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"## Configuration\\n\",\n    \"[_(back to toc)_](#Table-of-contents)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Paths:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"libsvm_data_remote_path = 'criteo/libsvm'\\n\",\n    \"local_runtime_path = 'criteo/runtime'\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"import os\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"libsvm_train_template = os.path.join(libsvm_data_remote_path, 'train', '{}')\\n\",\n    \"libsvm_test_template = os.path.join(libsvm_data_remote_path, 'test', '{}')\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"def ensure_directory_exists(path):\\n\",\n    \"    if not os.path.exists(path):\\n\",\n    \"        os.makedirs(path)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Samples to take:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"train_samples = [\\n\",\n    \"    10000, 30000,  # tens of thousands\\n\",\n    \"    100000, 300000,  # hundreds of thousands\\n\",\n    \"    1000000, 3000000,  # millions\\n\",\n    \"    10000000, 30000000,  # tens of millions\\n\",\n    \"    100000000, 300000000,  # hundreds of millions\\n\",\n    \"    1000000000, 3000000000,  # billions\\n\",\n    \"]\\n\",\n    \"\\n\",\n    \"test_samples = [1000000]\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Spark configuration and initialization:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"executor_instances = 64\\n\",\n    \"executor_cores = 4\\n\",\n    \"memory_per_core = 4\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"app_name = 'Criteo experiment'\\n\",\n    \"\\n\",\n    \"master = 'yarn'\\n\",\n    \"\\n\",\n    \"settings = {\\n\",\n    \"    'spark.network.timeout': '600',\\n\",\n    \"    \\n\",\n    \"    'spark.driver.cores': '16',\\n\",\n    \"    'spark.driver.maxResultSize': '16G',\\n\",\n    \"    'spark.driver.memory': '32G',\\n\",\n    \"    \\n\",\n    \"    'spark.executor.cores': str(executor_cores),\\n\",\n    \"    'spark.executor.instances': str(executor_instances),\\n\",\n    \"    'spark.executor.memory': str(memory_per_core * executor_cores) + 'G',\\n\",\n    \"    \\n\",\n    \"    'spark.speculation': 'true',\\n\",\n    \"    \\n\",\n    \"    'spark.yarn.queue': 'root.HungerGames',\\n\",\n    \"}\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true,\n    \"scrolled\": false\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from pyspark.sql import SparkSession\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"builder = SparkSession.builder\\n\",\n    \"\\n\",\n    \"builder.appName(app_name)\\n\",\n    \"builder.master(master)\\n\",\n    \"for k, v in settings.items():\\n\",\n    \"    builder.config(k, v)\\n\",\n    \"\\n\",\n    \"spark = builder.getOrCreate()\\n\",\n    \"sc = spark.sparkContext\\n\",\n    \"\\n\",\n    \"sc.setLogLevel('ERROR')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Logging:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"import sys\\n\",\n    \"import logging\\n\",\n    \"reload(logging)\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"handler = logging.StreamHandler(stream=sys.stdout)\\n\",\n    \"formatter = logging.Formatter('[%(asctime)s] %(message)s')\\n\",\n    \"handler.setFormatter(formatter)\\n\",\n    \"\\n\",\n    \"ensure_directory_exists(local_runtime_path)\\n\",\n    \"file_handler = logging.FileHandler(filename=os.path.join(local_runtime_path, 'mylog.log'), mode='a')\\n\",\n    \"file_handler.setFormatter(formatter)\\n\",\n    \"\\n\",\n    \"logger = logging.getLogger()\\n\",\n    \"logger.addHandler(handler)\\n\",\n    \"logger.addHandler(file_handler)\\n\",\n    \"logger.setLevel(logging.DEBUG)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"logger.info('Spark version: %s.', spark.version)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Plot measurements:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true,\n    \"scrolled\": false\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"import pandas\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"def extract_data_for_plotting(df, what):\\n\",\n    \"    return reduce(\\n\",\n    \"        lambda left, right: pandas.merge(left, right, how='outer', on='Train size'),\\n\",\n    \"        map(\\n\",\n    \"            lambda name: df[df.Engine == name][['Train size', what]].rename(columns={what: name}),\\n\",\n    \"            df.Engine.unique(),\\n\",\n    \"        ),\\n\",\n    \"    )   \\n\",\n    \"\\n\",\n    \"def plot_stuff(df, what, ylabel=None, **kwargs):\\n\",\n    \"    data = extract_data_for_plotting(df, what).set_index('Train size')\\n\",\n    \"    ax = data.plot(marker='o', figsize=(6, 6), title=what, grid=True, linewidth=2.0, **kwargs)  # xlim=(1e4, 4e9)\\n\",\n    \"    if ylabel is not None:\\n\",\n    \"        ax.set_ylabel(ylabel)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Let's name samples as their shortened \\\"engineering\\\" notation - 1e5 is 100k etc.:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"def sample_name(sample):\\n\",\n    \"    return str(sample)[::-1].replace('000', 'k')[::-1]\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"## Distributed training\\n\",\n    \"[_(back to toc)_](#Table-of-contents)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Loading of LibSVM data as Spark.ML dataset:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from pyspark.ml.linalg import SparseVector\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"def parse_libsvm_line_for_rf(line):\\n\",\n    \"    parts = line.split(' ')\\n\",\n    \"    label = int(parts[0])\\n\",\n    \"    indices, values = zip(*map(lambda s: s.split(':'), parts[1:]))\\n\",\n    \"    return (label, SparseVector(40, map(int, indices), map(int, values)))\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"task_splitting = 1  # tasks per core\\n\",\n    \"\\n\",\n    \"def load_ml_data_for_rf(template, sample):\\n\",\n    \"    path = template.format(sample_name(sample))\\n\",\n    \"    return sc.textFile(path).map(parse_libsvm_line_for_rf).toDF(['label', 'features']).repartition(executor_cores * executor_instances * task_splitting)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Evaluating a model:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from matplotlib import pyplot\\n\",\n    \"from sklearn.metrics import auc, log_loss, roc_curve\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"def calculate_roc(predictions):\\n\",\n    \"    labels, scores = zip(*predictions.rdd.map(lambda row: (row.label, row.probability[1])).collect())\\n\",\n    \"    fpr, tpr, _ = roc_curve(labels, scores)\\n\",\n    \"    roc_auc = auc(fpr, tpr)\\n\",\n    \"    ll = log_loss(labels, scores)\\n\",\n    \"    return fpr, tpr, roc_auc, ll\\n\",\n    \"\\n\",\n    \"def evaluate_model(name, model, test, train=None):\\n\",\n    \"    metrics = dict()\\n\",\n    \"    \\n\",\n    \"    figure = pyplot.figure(figsize=(6, 6))\\n\",\n    \"    ax = figure.gca()\\n\",\n    \"    ax.set_title('ROC - ' + name)\\n\",\n    \"    \\n\",\n    \"    if train is not None:\\n\",\n    \"        train_predictions = model.transform(train)\\n\",\n    \"        train_fpr, train_tpr, train_roc_auc, train_log_loss = calculate_roc(train_predictions)\\n\",\n    \"        \\n\",\n    \"        metrics['train_roc_auc'] = train_roc_auc\\n\",\n    \"        metrics['train_log_loss'] = train_log_loss\\n\",\n    \"        \\n\",\n    \"        ax.plot(train_fpr, train_tpr, linewidth=2.0, label='train (auc = {:.3f})'.format(train_roc_auc))\\n\",\n    \"    \\n\",\n    \"    test_predictions = model.transform(test)\\n\",\n    \"    test_fpr, test_tpr, test_roc_auc, test_log_loss = calculate_roc(test_predictions)\\n\",\n    \"    \\n\",\n    \"    metrics['test_roc_auc'] = test_roc_auc\\n\",\n    \"    metrics['test_log_loss'] = test_log_loss\\n\",\n    \"    \\n\",\n    \"    ax.plot(test_fpr, test_tpr, linewidth=2.0, label='test (auc = {:.3f})'.format(test_roc_auc))\\n\",\n    \"    \\n\",\n    \"    ax.plot([0.0, 1.0], [0.0, 1.0], linestyle='--', c='gray')\\n\",\n    \"    ax.legend()\\n\",\n    \"    pyplot.show()\\n\",\n    \"    \\n\",\n    \"    return metrics\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Models to work on:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from pyspark.ml.classification import (\\n\",\n    \"    RandomForestClassifier, \\n\",\n    \")\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"classifiers = {\\n\",\n    \"    'rf': RandomForestClassifier(featureSubsetStrategy='sqrt', impurity='entropy', minInstancesPerNode=3, maxBins=64, maxDepth=10, numTrees=160),\\n\",\n    \"}\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Monkey-patch RDDs and DataFrames for context persistence:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"import pyspark\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"def enter_method(self):\\n\",\n    \"    self.persist()\\n\",\n    \"\\n\",\n    \"def exit_method(self,exc_type, exc, traceback):\\n\",\n    \"    self.unpersist()\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"pyspark.sql.dataframe.DataFrame.__enter__ = enter_method\\n\",\n    \"pyspark.sql.dataframe.DataFrame.__exit__ = exit_method\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Do distributed training:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true,\n    \"scrolled\": false\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"import time\\n\",\n    \"\\n\",\n    \"from pyspark.ml.evaluation import BinaryClassificationEvaluator\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"test_sample = test_samples[-1]\\n\",\n    \"\\n\",\n    \"evaluator = BinaryClassificationEvaluator(labelCol='label', rawPredictionCol='probability', metricName='areaUnderROC')\\n\",\n    \"\\n\",\n    \"new_quality_data = []\\n\",\n    \"new_data_rows = []\\n\",\n    \"\\n\",\n    \"logger.info('Loading \\\"%s\\\" test samples for rf.', test_sample)\\n\",\n    \"test_df = load_ml_data_for_rf(libsvm_test_template, test_sample)\\n\",\n    \"with test_df:\\n\",\n    \"    logger.info('Loaded \\\"%s\\\" lines.', test_df.count())\\n\",\n    \"\\n\",\n    \"    for train_sample in train_samples:\\n\",\n    \"\\n\",\n    \"        logger.info('Working on \\\"%s\\\" train sample.', train_sample)\\n\",\n    \"\\n\",\n    \"        logger.info('Loading \\\"%s\\\" train samples for rf.', train_sample)\\n\",\n    \"        train_df = load_ml_data_for_rf(libsvm_train_template, train_sample)\\n\",\n    \"        with train_df:\\n\",\n    \"            logger.info('Loaded \\\"%s\\\" lines.', train_df.count())\\n\",\n    \"\\n\",\n    \"            for classifier_name, classifier in classifiers.items():\\n\",\n    \"\\n\",\n    \"                logger.info('Training a model \\\"%s\\\" on sample \\\"%s\\\".', classifier_name, train_sample)\\n\",\n    \"\\n\",\n    \"                start = time.time()\\n\",\n    \"                model = classifier.fit(train_df)\\n\",\n    \"                duration = time.time() - start\\n\",\n    \"\\n\",\n    \"                logger.info('Training a model \\\"%s\\\" on sample \\\"%s\\\" took \\\"%g\\\" seconds.', classifier_name, train_sample, duration)\\n\",\n    \"\\n\",\n    \"                logger.info('Evaluating the model \\\"%s\\\" trained on sample \\\"%s\\\".', classifier_name, train_sample)\\n\",\n    \"                metrics = evaluate_model(classifier_name + ' - ' + sample_name(train_sample), model, test_df, train=(train_df if train_sample <= 1000000 else None))\\n\",\n    \"\\n\",\n    \"                test_predictions = model.transform(test_df)\\n\",\n    \"                ml_metric_value = evaluator.evaluate(test_predictions)\\n\",\n    \"\\n\",\n    \"                logger.info(\\n\",\n    \"                    'For the model \\\"%s\\\" trained on sample \\\"%s\\\" metrics are: \\\"%s\\\"; ROC AUC calculated by Spark is \\\"%s\\\".',\\n\",\n    \"                    classifier_name,\\n\",\n    \"                    train_sample,\\n\",\n    \"                    metrics,\\n\",\n    \"                    ml_metric_value,\\n\",\n    \"                )\\n\",\n    \"\\n\",\n    \"                data_row = {\\n\",\n    \"                    'Train size': train_sample,\\n\",\n    \"                    'ROC AUC': metrics['test_roc_auc'],\\n\",\n    \"                    'Log loss': metrics['test_log_loss'],\\n\",\n    \"                    'Duration': duration,\\n\",\n    \"                    'Engine': classifier_name,\\n\",\n    \"                }\\n\",\n    \"                new_quality_data.append(data_row)\\n\",\n    \"                data_row_string = '\\\\t'.join(str(data_row[field]) for field in ['Engine', 'Train size', 'ROC AUC', 'Log loss', 'Duration'])\\n\",\n    \"                new_data_rows.append(data_row_string)\\n\",\n    \"                logger.info('Data row: \\\"%s\\\".', data_row_string)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Plot metrics:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true,\n    \"scrolled\": false\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"measurements_df = pandas.DataFrame(new_quality_data).sort_values(by=['Train size'])\\n\",\n    \"plot_stuff(measurements_df, 'ROC AUC', logx=True)\\n\",\n    \"plot_stuff(measurements_df, 'Log loss', logx=True, ylim=(0.135, 0.145))\\n\",\n    \"plot_stuff(measurements_df, 'Duration', loglog=True, ylabel='s')\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"for row in new_data_rows:\\n\",\n    \"    print(row)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"## End\\n\",\n    \"[_(back to toc)_](#Table-of-contents)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"Work done, stop Spark:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"collapsed\": true\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"spark.stop()\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"kernelspec\": {\n   \"display_name\": \"Python 3\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.6.1\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n"
  },
  {
    "path": "results/metrics.cluster.tsv",
    "content": "\"Engine\"\t\"Train size\"\t\"ROC AUC\"\t\"Log loss\"\t\"Train time\"\n\"lr\"\t10000\t0.577664202812\t0.191408441626\t50.9623498917\n\"lr\"\t30000\t0.593790859746\t0.181353099861\t87.1837468147\n\"lr\"\t100000\t0.612247160316\t0.169781474994\t148.152549982\n\"lr\"\t300000\t0.644818188231\t0.155259096662\t138.703317881\n\"lr\"\t1000000\t0.684101056031\t0.142522071064\t154.500109911\n\"lr\"\t3000000\t0.720519424179\t0.135748217546\t140.207423925\n\"lr\"\t10000000\t0.742150224051\t0.133061793307\t175.111939907\n\"lr\"\t30000000\t0.750851634783\t0.132163191494\t205.059295177\n\"lr\"\t100000000\t0.754105416714\t0.131909898252\t429.173339128\n\"lr\"\t300000000\t0.755291615978\t0.131809908334\t655.645493984\n\"lr\"\t1000000000\t0.755555957222\t0.131795697268\t5133.28028798\n\"lr\"\t3000000000\t0.755575083657\t0.131792462522\t5726.66465712\n\"rf\"\t10000\t0.666209405818\t0.141072504942\t162.25733614\n\"rf\"\t30000\t0.686325476035\t0.139292662004\t123.259371996\n\"rf\"\t100000\t0.698154160624\t0.138366116883\t67.557528019\n\"rf\"\t300000\t0.704939649921\t0.137753153698\t78.4612021446\n\"rf\"\t1000000\t0.7072363872\t0.137494568389\t105.36420989\n\"rf\"\t3000000\t0.707892500228\t0.13742745787\t154.24793601\n\"rf\"\t10000000\t0.708613003835\t0.13734362384\t518.322438955\n\"rf\"\t30000000\t0.708447321414\t0.137352474673\t665.707078934\n\"rf\"\t100000000\t0.708270614391\t0.137371388594\t1821.9798851\n\"rf\"\t300000000\t0.708382095174\t0.137347952598\t3362.20391607\n"
  },
  {
    "path": "results/metrics.lr_hash_size.tsv",
    "content": "\"Engine\"\t\"Train size\"\t\"ROC AUC\"\t\"Log loss\"\t\"Train time\"\n\"lr, 100k hashes\"\t10000\t0.577664202812\t0.191408441626\t50.9623498917\n\"lr, 100k hashes\"\t30000\t0.593790859746\t0.181353099861\t87.1837468147\n\"lr, 100k hashes\"\t100000\t0.612247160316\t0.169781474994\t148.152549982\n\"lr, 100k hashes\"\t300000\t0.644818188231\t0.155259096662\t138.703317881\n\"lr, 100k hashes\"\t1000000\t0.684101056031\t0.142522071064\t154.500109911\n\"lr, 100k hashes\"\t3000000\t0.720519424179\t0.135748217546\t140.207423925\n\"lr, 100k hashes\"\t10000000\t0.742150224051\t0.133061793307\t175.111939907\n\"lr, 100k hashes\"\t30000000\t0.750851634783\t0.132163191494\t205.059295177\n\"lr, 100k hashes\"\t100000000\t0.754105416714\t0.131909898252\t429.173339128\n\"lr, 30k hashes\"\t10000\t0.575027699783\t0.188237385947\t34.5243330002\n\"lr, 30k hashes\"\t30000\t0.594786944726\t0.174772368249\t47.6537959576\n\"lr, 30k hashes\"\t100000\t0.625299136967\t0.15797820297\t94.8742370605\n\"lr, 30k hashes\"\t300000\t0.669467206714\t0.144246165797\t106.976841927\n\"lr, 30k hashes\"\t1000000\t0.71062957893\t0.136591896049\t106.445657969\n\"lr, 30k hashes\"\t3000000\t0.732331332028\t0.134246693916\t117.792547941\n\"lr, 30k hashes\"\t10000000\t0.742112278546\t0.133347719357\t139.843713999\n\"lr, 30k hashes\"\t30000000\t0.744923892422\t0.133133669898\t171.719541073\n\"lr, 30k hashes\"\t100000000\t0.7463477576\t0.133017218565\t478.496304035\n"
  },
  {
    "path": "results/metrics.old.tsv",
    "content": "\"Engine\"\t\"Train size\"\t\"ROC AUC\"\t\"Log loss\"\t\"Train time\"\n\"vw\"\t10000\t0.609\t0.1501\t1\n\"vw\"\t30000\t0.632\t0.1492\t2\n\"vw\"\t100000\t0.653\t0.1461\t3\n\"vw\"\t300000\t0.679\t0.1429\t4\n\"vw\"\t1000000\t0.692\t0.1413\t5\n\"vw\"\t3000000\t0.707\t0.1387\t6\n\"vw\"\t10000000\t0.72\t0.1369\t7\n\"xgb\"\t10000\t0.65\t0.1472\t8\n\"xgb\"\t30000\t0.67\t0.1451\t9\n\"xgb\"\t100000\t0.683\t0.1442\t10\n\"xgb\"\t300000\t0.694\t0.14355\t11\n\"xgb\"\t1000000\t0.695\t0.14335\t12\n\"xgb\"\t3000000\t0.696\t0.14325\t13\n\"xgb\"\t10000000\t0.6965\t0.14323\t14\n\"xgb.ooc\"\t10000\t0.642\t0.1476\t15\n\"xgb.ooc\"\t30000\t0.672\t0.1446\t16\n\"xgb.ooc\"\t100000\t0.685\t0.144\t17\n\"xgb.ooc\"\t300000\t0.694\t0.1435\t18\n\"xgb.ooc\"\t1000000\t0.695\t0.1433\t19\n\"xgb.ooc\"\t3000000\t0.696\t0.1433\t20\n\"xgb.ooc\"\t10000000\t0.6964\t0.143231\t21\n"
  },
  {
    "path": "results/metrics.selection.tsv",
    "content": "\"Engine\"\t\"Train size\"\t\"ROC AUC\"\t\"Log loss\"\t\"Train time\"\nlr\t10000\t0.578177884844\t0.178971884565\t29.5085098743\nrf\t10000\t0.621504103494\t0.144342437741\t76.4821279049\ntree\t10000\t0.600422662988\t0.357273201261\t11.1679940224\nbayes\t10000\t0.509993205115\t0.818047776891\t1.48214101791\nlr\t30000\t0.606712654428\t0.16236564437\t35.3462071419\nrf\t30000\t0.643115035038\t0.142662388067\t92.7048139572\ntree\t30000\t0.617018579741\t0.165522218545\t11.9768800735\nbayes\t30000\t0.565664291131\t0.473385707556\t0.820657014847\nlr\t100000\t0.652313535925\t0.145698453913\t40.4066979885\nrf\t100000\t0.663411563927\t0.14115005143\t144.407087088\ntree\t100000\t0.638114905876\t0.149601338341\t12.5886089802\nbayes\t100000\t0.637296917738\t0.373915545595\t0.909769058228\nlr\t300000\t0.694017701134\t0.138406933493\t41.6715488434\nrf\t300000\t0.674058380306\t0.140104620859\t244.973959923\ntree\t300000\t0.641005555466\t0.142781786127\t16.7563710213\nbayes\t300000\t0.672807318246\t0.418186728992\t1.29778695107\nlr\t1000000\t0.719469466249\t0.135623586024\t46.4208109379\nrf\t1000000\t0.677311170308\t0.139673520936\t586.515081882\ntree\t1000000\t0.64388298082\t0.141953441838\t29.7467141151\nbayes\t1000000\t0.690009509821\t0.437564216119\t1.74085712433\nlr\t3000000\t0.729501417959\t0.134854378505\t52.4596869946\nrf\t3000000\t0.680087853763\t0.13939451653\t1414.78307986\ntree\t3000000\t0.642813675941\t0.141866601496\t65.7208981514\nbayes\t3000000\t0.697140532878\t0.450650398396\t2.11705684662\n"
  },
  {
    "path": "results/metrics.tsv",
    "content": "\"Engine\"\t\"Train size\"\t\"ROC AUC\"\t\"Log loss\"\t\"Train time\"\n\"vw\"\t10000\t0.617455161515\t0.144524706901\t15.17\n\"vw\"\t30000\t0.647287001387\t0.142233353612\t13.9\n\"vw\"\t100000\t0.683247769366\t0.139338887152\t16.62\n\"vw\"\t300000\t0.70142316924\t0.13745122263\t17.75\n\"vw\"\t1000000\t0.716287545509\t0.13588863014\t23.83\n\"vw\"\t3000000\t0.727931938776\t0.134788994538\t40.1\n\"vw\"\t10000000\t0.739548558371\t0.133521239905\t91.08\n\"vw\"\t30000000\t0.746322264127\t0.132553436306\t242.17\n\"vw\"\t100000000\t0.752882908682\t0.131795464547\t718.71\n\"vw\"\t300000000\t0.754807741333\t0.131569260518\t1989.99\n\"vw\"\t1000000000\t0.757005674498\t0.131394975718\t6431.0\n\"vw\"\t3000000000\t0.756123547995\t0.132255694498\t17798.0\n\"xgb\"\t10000\t0.663509743693\t0.142349739278\t1.56\n\"xgb\"\t30000\t0.682316321881\t0.140907061928\t4.58\n\"xgb\"\t100000\t0.696964467698\t0.139122124164\t14.86\n\"xgb\"\t300000\t0.714743656593\t0.136567639058\t45.52\n\"xgb\"\t1000000\t0.733393270406\t0.134198458139\t179.49\n\"xgb\"\t3000000\t0.744253560697\t0.13280500935\t610.78\n\"xgb\"\t10000000\t0.750706783469\t0.131929939622\t2541.02\n\"xgb\"\t30000000\t0.753289508009\t0.131529827949\t9065\n\"xgb\"\t100000000\t0.755224491108\t0.131270032458\t37105\n\"xgb.ooc\"\t10000\t0.663936721734\t0.142204298072\t6.52\n\"xgb.ooc\"\t30000\t0.684182538527\t0.140580030251\t24.2\n\"xgb.ooc\"\t100000\t0.697765852364\t0.138864711732\t87.48\n\"xgb.ooc\"\t300000\t0.714633362901\t0.136556600616\t264.64\n\"xgb.ooc\"\t1000000\t0.731054714952\t0.134485312106\t944.91\n\"xgb.ooc\"\t3000000\t0.742823599115\t0.132987550283\t2517.86\n\"xgb.ooc\"\t10000000\t0.750953801983\t0.131909674184\t5874\n\"xgb.ooc\"\t30000000\t0.753881106383\t0.131430669756\t14520\n\"xgb.ooc\"\t100000000\t0.755746234321\t0.131237777868\t44875\n\"lr\"\t10000\t0.577664202812\t0.191408441626\t50.9623498917\n\"lr\"\t30000\t0.593790859746\t0.181353099861\t87.1837468147\n\"lr\"\t100000\t0.612247160316\t0.169781474994\t148.152549982\n\"lr\"\t300000\t0.644818188231\t0.155259096662\t138.703317881\n\"lr\"\t1000000\t0.684101056031\t0.142522071064\t154.500109911\n\"lr\"\t3000000\t0.720519424179\t0.135748217546\t140.207423925\n\"lr\"\t10000000\t0.742150224051\t0.133061793307\t175.111939907\n\"lr\"\t30000000\t0.750851634783\t0.132163191494\t205.059295177\n\"lr\"\t100000000\t0.754105416714\t0.131909898252\t429.173339128\n\"lr\"\t300000000\t0.755291615978\t0.131809908334\t655.645493984\n\"lr\"\t1000000000\t0.755555957222\t0.131795697268\t5133.28028798\n\"lr\"\t3000000000\t0.755575083657\t0.131792462522\t5726.66465712\n\"rf\"\t10000\t0.666209405818\t0.141072504942\t162.25733614\n\"rf\"\t30000\t0.686325476035\t0.139292662004\t123.259371996\n\"rf\"\t100000\t0.698154160624\t0.138366116883\t67.557528019\n\"rf\"\t300000\t0.704939649921\t0.137753153698\t78.4612021446\n\"rf\"\t1000000\t0.7072363872\t0.137494568389\t105.36420989\n\"rf\"\t3000000\t0.707892500228\t0.13742745787\t154.24793601\n\"rf\"\t10000000\t0.708613003835\t0.13734362384\t518.322438955\n\"rf\"\t30000000\t0.708447321414\t0.137352474673\t665.707078934\n\"rf\"\t100000000\t0.708270614391\t0.137371388594\t1821.9798851\n\"rf\"\t300000000\t0.708382095174\t0.137347952598\t3362.20391607\n"
  },
  {
    "path": "results/vw_xgb.tsv",
    "content": "\"Engine\"\t\"Train size\"\t\"ROC AUC\"\t\"Log loss\"\t\"Train time\"\t\"Maximum memory\"\t\"CPU load\"\n\"vw\"\t10000\t0.617455161515\t0.144524706901\t15.17\t8615268352\t1.0\n\"vw\"\t30000\t0.647287001387\t0.142233353612\t13.9\t8615268352\t0.99\n\"vw\"\t100000\t0.683247769366\t0.139338887152\t16.62\t8615727104\t1.01\n\"vw\"\t300000\t0.70142316924\t0.13745122263\t17.75\t8616218624\t1.06\n\"vw\"\t1000000\t0.716287545509\t0.13588863014\t23.83\t8618024960\t1.2\n\"vw\"\t3000000\t0.727931938776\t0.134788994538\t40.1\t8615903232\t1.34\n\"vw\"\t10000000\t0.739548558371\t0.133521239905\t91.08\t8617746432\t1.48\n\"vw\"\t30000000\t0.746322264127\t0.132553436306\t242.17\t8616017920\t1.57\n\"vw\"\t100000000\t0.752882908682\t0.131795464547\t718.71\t8617304064\t1.62\n\"vw\"\t300000000\t0.754807741333\t0.131569260518\t1989.99\t8616005632\t1.67\n\"vw\"\t1000000000\t0.757005674498\t0.131394975718\t6431.0\t8617570304\t1.71\n\"vw\"\t3000000000\t0.756123547995\t0.132255694498\t17798.0\t8617238528\t1.78\n\"xgb\"\t10000\t0.663509743693\t0.142349739278\t1.56\t18403328\t8.4\n\"xgb\"\t30000\t0.682316321881\t0.140907061928\t4.58\t33456128\t10.63\n\"xgb\"\t100000\t0.696964467698\t0.139122124164\t14.86\t71958528\t12.06\n\"xgb\"\t300000\t0.714743656593\t0.136567639058\t45.52\t193576960\t13.07\n\"xgb\"\t1000000\t0.733393270406\t0.134198458139\t179.49\t617226240\t13.2\n\"xgb\"\t3000000\t0.744253560697\t0.13280500935\t610.78\t1828356096\t14.27\n\"xgb\"\t10000000\t0.750706783469\t0.131929939622\t2541.02\t6056378368\t14.12\n\"xgb\"\t30000000\t0.753289508009\t0.131529827949\t9065\t18139652096\t13.51\n\"xgb\"\t100000000\t0.755224491108\t0.131270032458\t37105\t60420395008\t10.78\n\"xgb.ooc\"\t10000\t0.663936721734\t0.142204298072\t6.52\t31752192\t3.71\n\"xgb.ooc\"\t30000\t0.684182538527\t0.140580030251\t24.2\t48721920\t3.68\n\"xgb.ooc\"\t100000\t0.697765852364\t0.138864711732\t87.48\t109694976\t3.72\n\"xgb.ooc\"\t300000\t0.714633362901\t0.136556600616\t264.64\t279904256\t4.09\n\"xgb.ooc\"\t1000000\t0.731054714952\t0.134485312106\t944.91\t852369408\t4.98\n\"xgb.ooc\"\t3000000\t0.742823599115\t0.132987550283\t2517.86\t1456574464\t5.79\n\"xgb.ooc\"\t10000000\t0.750953801983\t0.131909674184\t5874\t1921560576\t8.2\n\"xgb.ooc\"\t30000000\t0.753881106383\t0.131430669756\t14520\t2562379776\t9.95\n\"xgb.ooc\"\t100000000\t0.755746234321\t0.131237777868\t44875\t4807467008\t8.79\n"
  },
  {
    "path": "scripts/conversion/criteoToLibsvm.scala",
    "content": "def rowToLibsvm(row: org.apache.spark.sql.Row): String = {\n  0 until row.length flatMap {\n    case 0 => Some(row(0).toString)\n    case i if row(i) == null => None\n    case i => Some(i.toString + ':' + (if (i < 14) row(i) else java.lang.Long.parseLong(row(i).toString, 16)).toString)\n  } mkString \" \"\n}\n\ndef readDataFrame(path: String): org.apache.spark.sql.DataFrame = {\n  spark.read.option(\"header\", \"false\").option(\"inferSchema\", \"true\").option(\"delimiter\", \"\\t\").csv(path)\n}\n\ndef writeDataFrame(df: org.apache.spark.sql.DataFrame, path: String): Unit = {\n  df.rdd.map(rowToLibsvm).saveAsTextFile(path)\n}\n\ndef processDay(day: Int): Unit = {\n  println(s\"Processing of the day $day started\")\n  val inputPath = s\"criteo_1tb/plain/day_$day\"\n  println(s\"Loading data from $inputPath\")\n  val df = readDataFrame(inputPath)\n  val outputPath = s\"criteo_1tb/libsvm/day_$day.libsvm\"\n  println(s\"Saving data to $outputPath\")\n  writeDataFrame(df, outputPath)\n  println(s\"Processing of the day $day finished\")\n}\n\n\nprintln(\"Do '0 to 23 foreach processDay' to convert all data to LibSVM format.\")\n"
  },
  {
    "path": "scripts/conversion/libsvmToVw.scala",
    "content": "import java.security.MessageDigest\n\nimport org.apache.hadoop.fs.{FileSystem, Path => HadoopPath}\nimport org.apache.spark.rdd.RDD\n\n\ndef convertLine(row: String): String = {\n  val elements = row.split(' ')\n  val target  = elements.head.toInt * 2 - 1\n  (target.toString + \" |\") +: elements.tail.map(e => {\n    val es = e.split(':')\n    val index = es(0).toInt\n    if (index < 14) e\n    else es.mkString(\"_\")\n  }) mkString \" \"\n}\n\ndef md5[T](s: T) = {\n  MessageDigest\n    .getInstance(\"MD5\")\n    .digest(s.toString.getBytes)\n    .map(\"%02x\".format(_))\n    .mkString\n}\n\ndef convertFile(srcPath: String, dstPath: String) = {\n  sc\n    .textFile(srcPath)\n    .map(convertLine)\n    .zipWithIndex\n    .sortBy(z => md5(z._2))\n    .map(_._1)\n    .saveAsTextFile(dstPath)\n}\n\nval names = List(\"test\", \"train\")\n\ndef powTenAndTriple(n: Int): List[Long] = { val v = scala.math.pow(10, n).longValue; List(v, 3 * v) }\nval nums = (4 to 9 flatMap powTenAndTriple).toList\ndef numToString(num: Long): String = num.toString.reverse.replaceAll(\"000\", \"k\").reverse\n\nval fs = FileSystem.get(sc.hadoopConfiguration)\ndef fileExists(path: String): Boolean = fs.exists(new HadoopPath(path))\ndef removeFile(path: String): Unit = fs.delete(new HadoopPath(path))\n\ndef doDataConversion = {\n  for {\n    num <- nums\n    name <- names\n    numName = numToString(num)\n    srcPath = s\"criteo/libsvm/$name/$numName\"\n    dstPath = s\"criteo/vw/$name/$numName\"\n    if fileExists(srcPath)\n    if !fileExists(dstPath + \"/_SUCCESS\")\n  } {\n    println(s\"$srcPath -> $dstPath\")\n    removeFile(dstPath)\n    convertFile(srcPath, dstPath)\n  }\n}\n\n\nprintln(\"Use 'doDataConversion' to start data conversion.\")\n"
  },
  {
    "path": "scripts/conversion/sampleLibsvm.scala",
    "content": "type Data = (org.apache.spark.rdd.RDD[String], Long)\n\ndef sample(data: Data, n: Long): Data = {\n  val rdd = data._1\n  val count = data._2\n  val ratio = n.toDouble / count\n  val sampledRdd = rdd.sample(false, scala.math.min(1.0, ratio * 1.01), scala.util.Random.nextLong)\n  val exactSampledRdd = sampledRdd.zipWithIndex.filter { case (_, i) => i < n } map (_._1)\n  val exactCount = exactSampledRdd.count\n  (exactSampledRdd, exactCount)\n}\n\ndef load(path: String): (Data, Data) = {\n  val filesAndStats = 0 to 23 map {\n    case i => {\n      val dayPath = s\"$path/day_${i}.*\"\n      val rdd = sc.textFile(dayPath)\n      val count = rdd.count\n      (rdd, count)\n    }\n  }\n  val test = filesAndStats.last\n  val train = filesAndStats.init.reduce((a, b) => (a._1 union b._1, a._2 + b._2))\n  (train, test)\n}\n\nval fs = org.apache.hadoop.fs.FileSystem.get(sc.hadoopConfiguration)\n\nval numberOfParts = 1024\n\ndef writeSamples(data: Data, samples: List[Long], path: String, ext: String): Unit = samples foreach {\n  case n =>\n    val name = n.toString.reverse.replaceAll(\"000\", \"k\").reverse\n    val writePath = s\"$path/$name.$ext\"\n    val hadoopSuccessPath = new org.apache.hadoop.fs.Path(writePath + \"/_SUCCESS\")\n    if (fs.exists(hadoopSuccessPath)) {\n      println(s\"Data was already successfully written to $writePath, skipping.\")\n    } else {\n      val hadoopPath = new org.apache.hadoop.fs.Path(writePath)\n      println(s\"Removing $writePath.\")\n      fs.delete(hadoopPath)\n      println(\"Sampling data\")\n      val sampledData = sample(data, n)\n      println(s\"Writing ${sampledData._2} lines to $writePath.\")\n      sampledData._1.coalesce(numberOfParts).saveAsTextFile(writePath)\n    }\n}\n\ndef powTenAndTriple(n: Int): List[Long] = { val v = scala.math.pow(10, n).longValue; List(v, 3 * v) }\n\nval testSamples = List(1000000l)\nval trainSamples = (4 to 9 flatMap powTenAndTriple).toList\n\ndef processDataPersist(what: String): Unit = {\n  println(s\"Working with $what.\")\n\n  val dataPath = s\"criteo_1tb/$what\"\n  println(s\"Loading data from $dataPath.\")\n  val (train, test) = load(dataPath)\n  println(\"Data loaded.\")\n\n  def processDataSet(name: String, data: Data, samples: List[Long]): Unit = {\n    println(s\"Sampling $name to ${samples.mkString(\"[\", \", \", \"]\")} lines.\")\n    writeSamples(data, samples, s\"$dataPath/$name\", what)\n  }\n\n  test._1.persist\n  processDataSet(\"test\", test, testSamples)\n  test._1.unpersist(true)\n\n  train._1.persist\n  processDataSet(\"train\", train, trainSamples)\n  train._1.unpersist(true)\n\n  println(s\"Done with $what.\")\n}\n\ndef doDataPreparationLibSVM = {\n  processDataPersist(\"libsvm\")\n}\n\nprintln(\"Use 'doDataPreparationLibSVM' to start data preparation.\")\n"
  },
  {
    "path": "scripts/running/build_plots.sh",
    "content": "#!/usr/bin/env bash\n\npython plots.py -i metrics.old.tsv -n 'why_optimize' -c 'b,k,y'\npython plots.py -i metrics.selection.tsv -n 'cluster_selection' -c 'k,r,g,b'\npython plots.py -i vw_xgb.tsv -n 'local' -p -c 'b,k,y'\npython plots.py -i metrics.lr_hash_size.tsv -n 'lr_hash_size' -c 'r,g,b,k,y'\npython plots.py -i metrics.cluster.tsv -n 'cluster' -c 'r,g,b,k,y'\npython plots.py -i metrics.tsv -n 'local_and_cluster' -c 'r,g,b,k,y' -l '0.13,0.15'\n"
  },
  {
    "path": "scripts/running/measure.py",
    "content": "import sys\nfrom sklearn.metrics import (\n    auc,\n    log_loss,\n    roc_curve,\n)\n\n\nengine = sys.argv[1]\ntrain_file = sys.argv[2]\ntest_file = sys.argv[3]\n\nscores_file = test_file + '.predictions'\ntime_file = train_file + '.time'\n\n\ndef get_last_in_line(s):\n    return s.rstrip().split( )[-1]\n\ndef parse_elapsed_time(s):\n    return reduce(lambda a, b: a * 60 + b, map(float, get_last_in_line(s).split(':')))\n\ndef parse_max_memory(s):\n    return int(get_last_in_line(s)) * 1024\n\ndef parse_cpu(s):\n    return float(get_last_in_line(s).rstrip('%')) / 100\n\n\nelapsed = -1\nmemory = -1\ncpu = -1\n\nwith open(time_file, 'rb') as f:\n    for line in f:\n        if 'Elapsed (wall clock) time' in line:\n            elapsed = parse_elapsed_time(line)\n        elif 'Maximum resident set size' in line:\n            memory = parse_max_memory(line)\n        elif 'Percent of CPU' in line:\n            cpu = parse_cpu(line)\n\n\nwith open(test_file, 'rb') as f:\n    labels = [line.rstrip().split(' ')[0] == '1' for line in f]\n\nwith open(scores_file, 'rb') as f:\n    scores = [float(line.rstrip().split(' ')[0]) for line in f]\n\nfpr, tpr, _ = roc_curve(labels, scores)\nroc_auc = auc(fpr, tpr)\nll = log_loss(labels, scores)\n\n\ntry:\n    train_size = int(train_file.split('/')[-1].split('.')[2].replace('k', '000'))\nexcept:\n    train_size = 0\n\nprint '\\t'.join(map(str, [engine, train_size, roc_auc, ll, elapsed, memory, cpu]))\n"
  },
  {
    "path": "scripts/running/plots.py",
    "content": "from __future__ import print_function\n\nimport argparse\nimport re\n\nimport cycler\nimport pandas\n\nfrom matplotlib import pyplot\n\n\ndef extract_data_for_plotting(df, what):\n    return reduce(\n        lambda left, right: pandas.merge(\n            left,\n            right,\n            how='outer',\n            on='Train size',\n        ),\n        map(\n            lambda name: (\n                df[df.Engine == name][['Train size', what]]\n                .rename(columns={what: name})\n            ),\n            df.Engine.unique(),\n        ),\n    )\n\n\ndef plot_stuff(df, what, ylabel=None, **kwargs):\n    data = extract_data_for_plotting(df, what).set_index('Train size')\n    ax = data.plot(\n        figsize=(6, 6),\n        title=what,\n        grid=True,\n        linewidth=2.0,\n        marker='o',\n        **kwargs\n    )\n    ax.legend(loc='best')\n    if ylabel is not None:\n        ax.set_ylabel(ylabel)\n    ax.grid(which='major', linestyle='-')\n    ax.grid(which='minor', linestyle=':')\n\n    what_normalized = re.sub(r'\\s', '_', what).lower()\n\n    if experiment_name is not None:\n        ax.get_figure().savefig(what_normalized + '.' + experiment_name + '.png')\n    else:\n        ax.get_figure().savefig(what_normalized + '.png')\n\n\nparser = argparse.ArgumentParser()\n\nparser.add_argument('-i', '--input', type=str, default='metrics.tsv',\n                    help='input file')\n\nparser.add_argument('-n', '--name', type=str,\n                    help='experiment name')\n\nparser.add_argument('-p', '--perf', action='store_true',\n                    help='build perf graphs')\n\nparser.add_argument('-c', '--colors', type=str,\n                    help='color cycle')\n\nparser.add_argument('-l', '--logloss',\n                    help='log loss scale')\n\nargs = parser.parse_args()\n\n\nmetrics_file = args.input\nexperiment_name = args.name\nperf_graphs = args.perf\n\nif args.colors is not None:\n    color_cycle = args.colors.split(',')\n    pyplot.rc('axes', prop_cycle=(cycler.cycler('color', color_cycle)))\n\ndf = (\n    pandas\n    .read_csv(metrics_file, sep='\\t')\n    .sort_values(by=['Engine', 'Train size'])\n)\n\nplot_stuff(df, 'ROC AUC', logx=True)\n\nif args.logloss is not None:\n    ll_from, ll_to = map(float, args.logloss.split(','))\n    plot_stuff(df, 'Log loss', logx=True, ylim=(ll_from, ll_to))\nelse:\n    plot_stuff(df, 'Log loss', logx=True)\n\nplot_stuff(df, 'Train time', loglog=True, ylabel='s')\n\nif perf_graphs:\n    plot_stuff(df, 'Maximum memory', loglog=True, ylabel='bytes')\n    plot_stuff(df, 'CPU load', logx=True)\n"
  },
  {
    "path": "scripts/running/run.sh",
    "content": "#!/usr/bin/env bash\n\n\nDATA_PREFIX=\"./data\"\n\ntest_vw=\"${DATA_PREFIX}/data.test.1kk.vw\"\ntest_xgb=\"${DATA_PREFIX}/data.test.1kk.libsvm\"\n\nfor train_num in 10k 30k 100k 300k 1kk 3kk 10kk 30kk 100kk 300kk 1kkk 3kkk; do\n    echo \" * * * Train size ${train_num} lines * * *\"\n\n    train_vw=\"${DATA_PREFIX}/data.train.${train_num}.vw\"\n    train_xgb=\"${DATA_PREFIX}/data.train.${train_num}.libsvm\"\n\n    echo \"Running VW with train ${train_vw} and ${test_vw}\"\n    ./vw.sh \"${train_vw}\" \"${test_vw}\"\n\n    echo \"Running XGBoost with train ${train_xgb} and ${test_xgb}\"\n    ./xgb.sh \"${train_xgb}\" \"${test_xgb}\"\n\n    echo \"Running XGBoost (out-of-core) with train ${train_xgb} and ${test_xgb}\"\n    ./xgb.ooc.sh \"${train_xgb}\" \"${test_xgb}\"\ndone\n"
  },
  {
    "path": "scripts/running/vw.conf",
    "content": "-b          29\n-l          0.3\n--initial_t 1\n--decay_learning_rate   0.5\n--power_t   0.5\n--l1        1e-15\n--l2        0\n"
  },
  {
    "path": "scripts/running/vw.sh",
    "content": "#!/usr/bin/env bash\n\n\nTRAIN=\"${1}\"\nTEST=\"${2}\"\n\nif [[ \"${TRAIN}\" == \"\" || \"${TEST}\" == \"\" ]]; then\n    echo \"Usage: $0 train test\"\n    exit 1\nfi\n\nTIME=\"${TRAIN}.time\"\nMODEL=\"${TRAIN}.model\"\nPREDICTIONS=\"${TEST}.predictions\"\n\n\nVW_OPTS=($(cat vw.conf))\necho VW_OPTS = \"${VW_OPTS[@]}\"\n\n/usr/local/bin/time -v --output=\"${TIME}\" \\\n    vw83 --link=logistic --loss_function=logistic -d \"${TRAIN}\" -f \"${MODEL}\" \"${VW_OPTS[@]}\"\n\nvw83 -i \"${MODEL}\" --loss_function=logistic -t -d \"${TEST}\" -p \"${PREDICTIONS}\"\n\n\nMETRICS=\"metrics.tsv\"\nif ! [[ -e ${METRICS} ]]; then\n    echo -e \"Engine\\tTrain size\\tROC AUC\\tLog loss\\tTrain time\\tMaximum memory\\tCPU load\" | tee \"${METRICS}\"\nfi\n\npython measure.py \"vw\" \"${TRAIN}\" \"${TEST}\" | tee -a \"${METRICS}\"\n\nrm \"${TIME}\" \"${PREDICTIONS}\"\n"
  },
  {
    "path": "scripts/running/xgb.conf",
    "content": "booster = gbtree\nobjective = binary:logistic\nnthread = 12\neval_metric = logloss\n\nmax_depth =  7\nnum_round =  200\neta =  0.2\ngamma =  0.4\n\nsubsample =  0.8\ncolsample_bytree =  0.8\nmin_child_weight =  20\n\nalpha =  3\nlambda =  100\n"
  },
  {
    "path": "scripts/running/xgb.ooc.sh",
    "content": "#!/usr/bin/env bash\n\n\nTRAIN=\"${1}\"\nTEST=\"${2}\"\n\nTRAIN_OOC=\"${TRAIN}#${TRAIN}.cache\"\n\nif [[ \"${TRAIN}\" == \"\" || \"${TEST}\" == \"\" ]]; then\n    echo \"Usage: $0 train test\"\n    exit 1\nfi\n\nTIME=\"${TRAIN}.time\"\nMODEL=\"${TRAIN}.ooc.model\"\nPREDICTIONS=\"${TEST}.predictions\"\n\n\n/usr/local/bin/time -v --output=\"${TIME}\" \\\n    xgboost xgb.conf data=\"${TRAIN_OOC}\" model_out=\"${MODEL}\"\n\nxgboost xgb.conf task=pred test:data=\"${TEST}\" model_in=\"${MODEL}\" name_pred=\"${PREDICTIONS}\"\n\n\nMETRICS=\"metrics.tsv\"\nif ! [[ -e ${METRICS} ]]; then\n    echo -e \"Engine\\tTrain size\\tROC AUC\\tLog loss\\tTrain time\\tMaximum memory\\tCPU load\" | tee \"${METRICS}\"\nfi\n\npython measure.py \"xgb.ooc\" \"${TRAIN}\" \"${TEST}\" | tee -a \"${METRICS}\"\n\nrm \"${TIME}\" \"${PREDICTIONS}\" \"${TRAIN}.cache\"* \"${TEST}.buffer\"\n"
  },
  {
    "path": "scripts/running/xgb.sh",
    "content": "#!/usr/bin/env bash\n\n\nTRAIN=\"${1}\"\nTEST=\"${2}\"\n\nif [[ \"${TRAIN}\" == \"\" || \"${TEST}\" == \"\" ]]; then\n    echo \"Usage: $0 train test\"\n    exit 1\nfi\n\nTIME=\"${TRAIN}.time\"\nMODEL=\"${TRAIN}.model\"\nPREDICTIONS=\"${TEST}.predictions\"\n\n\n/usr/local/bin/time -v --output=\"${TIME}\" \\\n    xgboost xgb.conf data=\"${TRAIN}\" model_out=\"${MODEL}\"\n\nxgboost xgb.conf task=pred test:data=\"${TEST}\" model_in=\"${MODEL}\" name_pred=\"${PREDICTIONS}\"\n\n\nMETRICS=\"metrics.tsv\"\nif ! [[ -e ${METRICS} ]]; then\n    echo -e \"Engine\\tTrain size\\tROC AUC\\tLog loss\\tTrain time\\tMaximum memory\\tCPU load\" | tee \"${METRICS}\"\nfi\n\npython measure.py \"xgb\" \"${TRAIN}\" \"${TEST}\" | tee -a \"${METRICS}\"\n\nrm \"${TIME}\" \"${PREDICTIONS}\" \"${TRAIN}.buffer\" \"${TEST}.buffer\"\n"
  }
]