Repository: rambler-digital-solutions/criteo-1tb-benchmark
Branch: master
Commit: cc62a90f1386
Files: 22
Total size: 98.4 KB
Directory structure:
gitextract_hh2ek1q9/
├── README.md
├── notebooks/
│ ├── experiment_local.ipynb
│ ├── experiment_spark_lr.ipynb
│ └── experiment_spark_rf.ipynb
├── results/
│ ├── metrics.cluster.tsv
│ ├── metrics.lr_hash_size.tsv
│ ├── metrics.old.tsv
│ ├── metrics.selection.tsv
│ ├── metrics.tsv
│ └── vw_xgb.tsv
└── scripts/
├── conversion/
│ ├── criteoToLibsvm.scala
│ ├── libsvmToVw.scala
│ └── sampleLibsvm.scala
└── running/
├── build_plots.sh
├── measure.py
├── plots.py
├── run.sh
├── vw.conf
├── vw.sh
├── xgb.conf
├── xgb.ooc.sh
└── xgb.sh
================================================
FILE CONTENTS
================================================
================================================
FILE: README.md
================================================
# Criteo 1 TiB benchmark
## Table of contents
* [Introduction](#introduction)
* [Task and data](#task-and-data)
* [Algorithms](#algorithms)
* [Setup](#setup)
* [Experiment layout](#experiment-layout)
* [Hyperparameter optimization](#hyperparameter-optimization)
* [Data format for Spark.ML](#data-format-for-sparkml)
* [Code](#code)
* [Results](#results)
* [Local training - Vowpal Wabbit & XGBoost](#local-training---vowpal-wabbit--xgboost)
* [Distributed training - Spark.ML](#distributed-training---sparkml)
* [Distributed training - Time vs. Cores](#distributed-training---time-vs-cores)
* [Comparison of local vs. remote](#comparison-of-local-vs-remote)
* [Conclusion](#conclusion)
* [Resources](#resources)
## Introduction
[_(back to toc)_](#table-of-contents)
This project is a minimal benchmark of applicability of several implementations of machine learning algorithms to training on big data. Our main focus is [Spark.ML](http://spark.apache.org/mllib/) and how it compares to commonly used single-node machine learning tools Vowpal Wabbit and XGBoost in terms of scaling to terabyte (billions of lines) train data. Quick web search shows that many people tested Spark but not on tasks requiring a cluster so they are mostly single-node tests.
This project is inspired by https://github.com/szilard/benchm-ml but is focused on training models on billions of lines of train data, including Spark.ML in multinode cluster environment.
## Task and data
[_(back to toc)_](#table-of-contents)
Our target application is prediction of click-through ratio (CTR) of banners in online advertising. [Criteo released](http://labs.criteo.com/2015/03/criteo-releases-its-new-dataset/) an industry-standard open dataset which represents banner impressions in online advertising during the timespan of 24 days. It is more than 1 terabyte in size and consists of more than 4 billion lines of data. Each line represents a banner impression and contains 40 columns separated by tabulation:
- the first column is a label - {0, 1} - 1 meaning the banner was clicked and 0 otherwise;
- 13 numeric columns;
- 26 categorical columns with categories being 32-bit hashes.
This is how it looks like:
All the data except the last day was concatenated and sampled into training sets of 10ⁿ and 3×10ⁿ lines with `n ∈ {4, 5, ..., 9}` (i.e. train samples' sizes are 10k, 30k, 100k, ..., 1kkk, 3kkk lines). The last day was used for testing - a sample of one million lines was taken from it. All samples were converted to
- LibSVM format for training XGBoost models and as a source for the transformation to Spark.ML DataFrame;
- Vowpal Wabbit data format.
Data for Spark.ML models was processed on-the-fly from LibSVM format:
- into a dataset of tuples of "label" (integer) and "features" (SparseVector of size 10⁵ using [hashing trick](https://en.wikipedia.org/wiki/Feature_hashing#Feature_vectorization_using_the_hashing_trick) for all features) for Spark.ML LogisticRegression;
- into a dataset of tuples of "label" (integer) and "features" (SparseVector of size 39 taken as-is from corresponding columns, see below) for Spark.ML RandomForestClassifier.
## Algorithms
[_(back to toc)_](#table-of-contents)
Historically, we make use of [Vowpal Wabbit](https://github.com/JohnLangford/vowpal_wabbit) and [XGBoost](https://github.com/dmlc/xgboost) exploiting "Local train + Distributed apply" scenario. Our task was to run a performance test of our currently used approach and Spark.ML library algorithms.
We used the following non-distributed algorithms:
- Vowpal Wabbit - it implements logistic regression with a hashing trick and reads the data only once never keeping more than one sample in memory (it is an out-of-core implementation);
- in-memory XGBoost - gradient-boosted trees implementation that (by default) loads the whole data into memory (which is faster than multiple reads from disk, but we are limited in size by machine memory);
- out-of-core XGBoost - a variant of XGBoost training which uses an on-disk cache; this is slower (compared to the in-memory variant) but potentially we can train on the data limited in size only by the size of HDD.
Spark.ML contains following classification algorithms:
- [LogisticRegression](http://spark.apache.org/docs/latest/ml-classification-regression.html#logistic-regression),
- [RandomForestClassifier](http://spark.apache.org/docs/latest/ml-classification-regression.html#random-forest-classifier),
- [NaiveBayes](http://spark.apache.org/docs/latest/ml-classification-regression.html#naive-bayes),
- [DecisionTreeClassifier](http://spark.apache.org/docs/latest/ml-classification-regression.html#decision-tree-classifier),
- [GBTClassifier](http://spark.apache.org/docs/latest/ml-classification-regression.html#gradient-boosted-tree-classifier),
- [MultilayerPerceptronClassifier](http://spark.apache.org/docs/latest/ml-classification-regression.html#multilayer-perceptron-classifier).
Our preliminary research shows that four of the algorithms are not well-suited for our task of CTR prediction:
- NaiveBayes provides significantly worse logistic loss (which is an essential metric of CTR models' quality) than all other models;
- DecisionTreeClassifier suffers in quality in comparison to the RandomForestClassifier but still requires roughly the same amount of time to train;
- GBTClassifier (Spark.ML implementation of gradient-boosted trees) and MultilayerPerceptronClassifier do not support prediction of probabilities that are required by the task (these two models are not shown on graphs above).
  
Thus we use only LogisticRegression and RandomForestClassifier for our testing purposes.
## Setup
[_(back to toc)_](#table-of-contents)
Local models were trained on a 12-core (24-thread) machine with 128 GiB of memory. Distributed training was performed on our production cluster (total capacity is approximately 2000 cores and 10 TiB of memory); for the experiment a small part of resources has been allocated - 256 cores and 1 TiB of memory for training on datasets upto 300 million of lines and 512 cores and 2 TiB of memory for training on one billion and 3 billion lines of train data. 4 cores and 16 TiB of memory per Spark executor was used.
For the experiment we used Vowpal Wabbit 8.3.0, XGBoost 0.4 and Spark 2.1.0 running on a Hadoop 2.6 cluster (using YARN as a cluster manager).
## Experiment layout
### Hyperparameter optimization
[_(back to toc)_](#table-of-contents)
Our first idea was to skip models' hyperparameters optimization completely, but unfortunately XGBoost's default hyperparameters are not good enough for training even on million lines of data - the default number of trees is only 10, and it hits the ceiling quite soon:
 
These figures reminded us that production usage of any machine learning model is associated with optimization of its hyperparameters, and in our experiment we should do the same. For optimization of models' hyperparameters (including Spark.ML ones) we used the million-line sample of train data and 5-fold cross validation for metric (log loss) averaging.
### Data format for Spark.ML
[_(back to toc)_](#table-of-contents)
We tried to use [one-hot-encoding](https://www.quora.com/What-is-one-hot-encoding-and-when-is-it-used-in-data-science) of categorical features, but due to very large number of unique values it turned out to be very time and memory consuming, so for Spark.ML we decided to try the hashing trick. Spark.ML LogisticRegression was trained using this approach. We sticked to hashing space of 10⁵ hashes as it turned out to give about the same quality as VW on large samples. Taking less hashes usually leads to better quality on smaller data (because of less overfitting) and worse quality on bigger data (because some patterns in data are consumed by collisions in hashing space):
 
RandomForestClassifier was very slow to train even with a thousand hashes, so we used "as-is" format for it:
- all numeric features were converted to elements of SparseVector as-is;
- all categorical features were converted to elements of SparseVector by interpreting the hashes as 32 bit numbers.
### Code
[_(back to toc)_](#table-of-contents)
All work was performed in Jupyter notebooks in Python. Notebooks:
- [experiment_local.ipynb](notebooks/experiment_local.ipynb) was used for preparing the data and training of the local models;
- [experiment_spark_lr.ipynb](notebooks/experiment_spark_lr.ipynb) and [experiment_spark_rf.ipynb](notebooks/experiment_spark_rf.ipynb) for training Spark.ML LogisticRegression and RandomForestClassifier accordingly.
## Results
### Local training - Vowpal Wabbit & XGBoost
[_(back to toc)_](#table-of-contents)
    
Some observations:
- our main concern about an out-of-core training of XGBoost was that it would not produce the same quality as its in-memory variant due to the approximate splitting algorithm; however, in-memory XGBoost and out-of-core XGBoost turned out to provide about the same level of quality, but out-of-core variant is about an order of magnitude slower;
- in-memory XGBoost is about an order of magnitude slower than Vowpal Wabbit on the same amount of train data;
- Vowpal Wabbit was able to give about the same quality as XGBoost trained on an order of magnitude smaller sample.
### Distributed training - Spark.ML
[_(back to toc)_](#table-of-contents)
  
We made the following conclusions:
- RandomForestClassifier is quite slow, and it is even slower when the data consists of large vectors;
- LogisticRegression is hard to set up for smaller samples and for bigger samples at the same time - either it overfits on small data or it cannot extract patterns due to more aggressive hashing trick.
### Distributed training - Time vs. Cores
[_(back to toc)_](#table-of-contents)
To check how model training scales to multi-core setup we made a quick test where we increased the number of cores and measured training time for every step. To make it fast we used a 10⁷ sample of train data and checked training time for a number of cores from 5 to 50 in steps of 5. In order to eliminate the uncertainty brought forth by running the test in parallel with production tasks, for this test we created a standalone Spark cluster running on three machines with a total of ≈50 cores and ≈200 GiB of memory.
Training time dropped quite fast when we went from 5 to 15 cores but slowed down afterwards and completely ceased to improve by the mark of 40 cores (even growing a little on transition from 40 to 45 cores). The main idea we have extracted from this figure is that one should not increase amount of resources beyond minimum required, so that work distribution and aggregation overhead would be cheaper than potential improvement of speed by parallelization.
### Comparison of local vs. remote
[_(back to toc)_](#table-of-contents)
  
We can see that:
- on large datasets (100 million of lines and more) Spark.ML is faster than both Vowpal Wabbit and XGBoost; maybe it is possible to make it even faster by finding the best cluster setup for each size of training sample (we had not done this work);
- however it is slow when working with large vectors - steps should be taken in order to find a balance between quality and speed;
- for small tasks Spark introduces overhead that can be more expensive than it is possible to gain by computing the task in parallel (well, this is true for parallel computing in general).
## Conclusion
[_(back to toc)_](#table-of-contents)
The best quality measured by logarithmic loss (which is a metric of choice for CTR prediction) was achived using XGBoost - no matter in-memory or out-of-core, as they both seem to be equal in quality - on a sample smaller in size than other models required for the same level of quality.
However XGBoost is very slow on big samples in out-of-core setup thus it was not rational to test it on a 300kk sample and above (training the in-memory setup on large samples was also not possible due to memory restrictions).
The highest ROC AUC was reached by Vowpal Wabbit on one-billion-line train sample, strangely decreasing in quality by three-billion-line sample.
Spark.ML LogisticRegression is quite close in quality to Vowpal Wabbit, and maybe it can be made even better by increasing the feature space size (which is 100k hashes in current setup).
Spark.ML LogisticRegression appeared to be considerably faster than VW on billion-line samples and maybe it can be made even faster by optimizing the allocated resources.
Spark.ML RandomForestClassifier stopped increasing in quality quite early and it is also quite slow.
## Resources
[_(back to toc)_](#table-of-contents)
Results in table format can be found [here](results). Scala scripts used for faster conversion and sampling can be found [here](scripts/conversion) - these scripts can be used from [spark-shell](http://spark.apache.org/docs/latest/quick-start.html#basics) using `:load` command. Scripts for running VW & XGBoost and plotting outside of the Jupyter notebooks can be found [here](scripts/running).
================================================
FILE: notebooks/experiment_local.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Criteo 1 TiB benchmark\n",
"\n",
"In this experiment we will evalutate a number of machine learning tools on a varying size of train data to determine how fast they learn, how much memory they consume etc.\n",
"\n",
"We will assess Vowpal Wabbit and XGBoost in local mode, and Spark.ML models in cluster mode.\n",
"\n",
"We will use terabyte click logs released by Criteo and sample needed amount of data from them.\n",
"\n",
"This instance of experiment notebook focuses on data preparation and training VW & XGBoost locally.\n",
"\n",
"Let's go!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Table of contents\n",
"\n",
"* [Configuration](#Configuration)\n",
"* [Data preparation](#Data-preparation)\n",
" * [Criteo → LibSVM](#Criteo-→-LibSVM)\n",
" * [LibSVM → Train and test (sampling)](#LibSVM-→-Train-and-test-(sampling%29)\n",
" * [LibSVM train and test → VW train and test](#LibSVM-train-and-test-→-VW-train-and-test)\n",
" * [Local data](#Local-data)\n",
"* [Local training](#Local-training)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"%load_ext autotime\n",
"%matplotlib inline\n",
"\n",
"from __future__ import print_function"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Configuration\n",
"[_(back to toc)_](#Table-of-contents)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Paths:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"criteo_data_remote_path = 'criteo/plain'\n",
"libsvm_data_remote_path = 'criteo/libsvm'\n",
"vw_data_remote_path = 'criteo/vw'\n",
"\n",
"local_data_path = 'criteo/data'\n",
"local_results_path = 'criteo/results'\n",
"local_runtime_path = 'criteo/runtime'"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"\n",
"\n",
"criteo_day_template = os.path.join(criteo_data_remote_path, 'day_{}')\n",
"libsvm_day_template = os.path.join(libsvm_data_remote_path, 'day_{}')\n",
"vw_day_template = os.path.join(vw_data_remote_path, 'day_{}')\n",
"\n",
"libsvm_train_template = os.path.join(libsvm_data_remote_path, 'train', '{}')\n",
"libsvm_test_template = os.path.join(libsvm_data_remote_path, 'test', '{}')\n",
"vw_train_template = os.path.join(vw_data_remote_path, 'train', '{}')\n",
"vw_test_template = os.path.join(vw_data_remote_path, 'test', '{}')\n",
"\n",
"local_libsvm_test_template = os.path.join(local_data_path, 'data.test.{}.libsvm')\n",
"local_libsvm_train_template = os.path.join(local_data_path, 'data.train.{}.libsvm')\n",
"local_vw_test_template = os.path.join(local_data_path, 'data.test.{}.vw')\n",
"local_vw_train_template = os.path.join(local_data_path, 'data.train.{}.vw')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def ensure_directory_exists(path):\n",
" if not os.path.exists(path):\n",
" os.makedirs(path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Days to work on:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"days = list(range(0, 23 + 1))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Samples to take:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"train_samples = [\n",
" 10000, 30000, # tens of thousands\n",
" 100000, 300000, # hundreds of thousands\n",
" 1000000, 3000000, # millions\n",
" 10000000, 30000000, # tens of millions\n",
" 100000000, 300000000, # hundreds of millions\n",
" 1000000000, 3000000000, # billions\n",
"]\n",
"test_samples = [1000000]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Spark configuration and initialization:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"total_cores = 256"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"executor_cores = 4\n",
"executor_instances = total_cores / executor_cores\n",
"memory_per_core = 4"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"app_name = 'Criteo experiment'\n",
"\n",
"master = 'yarn'\n",
"\n",
"settings = {\n",
" 'spark.network.timeout': '600',\n",
" \n",
" 'spark.driver.cores': '16',\n",
" 'spark.driver.maxResultSize': '16G',\n",
" 'spark.driver.memory': '32G',\n",
" \n",
" 'spark.executor.cores': str(executor_cores),\n",
" 'spark.executor.instances': str(executor_instances),\n",
" 'spark.executor.memory': str(memory_per_core * executor_cores) + 'G',\n",
" \n",
" 'spark.speculation': 'true',\n",
" 'spark.yarn.queue': 'root.HungerGames',\n",
"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"from pyspark.sql import SparkSession\n",
"\n",
"\n",
"builder = SparkSession.builder\n",
"\n",
"builder.appName(app_name)\n",
"builder.master(master)\n",
"for k, v in settings.items():\n",
" builder.config(k, v)\n",
"\n",
"spark = builder.getOrCreate()\n",
"sc = spark.sparkContext\n",
"\n",
"sc.setLogLevel('ERROR')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Logging:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import sys\n",
"import logging\n",
"reload(logging)\n",
"\n",
"\n",
"handler = logging.StreamHandler(stream=sys.stdout)\n",
"formatter = logging.Formatter('[%(asctime)s] %(message)s')\n",
"handler.setFormatter(formatter)\n",
"\n",
"ensure_directory_exists(local_runtime_path)\n",
"file_handler = logging.FileHandler(filename=os.path.join(local_runtime_path, 'mylog.log'), mode='a')\n",
"file_handler.setFormatter(formatter)\n",
"\n",
"logger = logging.getLogger()\n",
"logger.addHandler(handler)\n",
"logger.addHandler(file_handler)\n",
"logger.setLevel(logging.INFO)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"logger.info('Spark version: %s.', spark.version)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Data preparation\n",
"[_(back to toc)_](#Table-of-contents)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Poor man's HDFS API:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def hdfs_exists(path):\n",
" l = !hadoop fs -ls $path 2>/dev/null\n",
" return len(l) != 0\n",
"\n",
"def hdfs_success(path):\n",
" return hdfs_exists(os.path.join(path, '_SUCCESS'))\n",
"\n",
"def hdfs_delete(path, recurse=False):\n",
" if recurse:\n",
" _ = !hadoop fs -rm -r $path\n",
" else:\n",
" _ = !hadoop fs -rm $path\n",
"\n",
"def hdfs_get(remote_path, local_path):\n",
" remote_path_glob = os.path.join(remote_path, 'part-*')\n",
" _ = !hadoop fs -cat $remote_path_glob >$local_path"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Load RDDs from one place and save them to another converted:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def convert_chunked_data(input_path_template, output_path_template, chunks, load_rdd, convert_row, transform_rdd=None):\n",
" for chunk in chunks:\n",
" input_path = input_path_template.format(chunk)\n",
" output_path = output_path_template.format(chunk)\n",
"\n",
" if hdfs_success(output_path):\n",
" logger.info('Chunk \"%s\" is already converted and saved to \"%s\", skipping.', chunk, output_path)\n",
" continue\n",
"\n",
" logger.info('Reading chunk \"%s\" data from \"%s\".', chunk, input_path)\n",
" rdd = load_rdd(input_path)\n",
"\n",
" if hdfs_exists(output_path):\n",
" logger.info('Cleaning \"%s\".', output_path)\n",
" hdfs_delete(output_path, recurse=True)\n",
"\n",
" logger.info('Processing and saving to \"%s\".', output_path)\n",
" rdd = rdd.map(convert_row)\n",
" \n",
" if transform_rdd is not None:\n",
" rdd = transform_rdd(rdd)\n",
" \n",
" rdd.saveAsTextFile(output_path)\n",
"\n",
" logger.info('Done with chunk \"%s\".', chunk)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Criteo → LibSVM\n",
"[_(back to toc)_](#Table-of-contents)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Criteo RDD is actually a DataFrame:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def load_criteo_rdd(path):\n",
" return (\n",
" spark\n",
" .read\n",
" .option('header', 'false')\n",
" .option('inferSchema', 'true')\n",
" .option('delimiter', '\\t')\n",
" .csv(path)\n",
" .rdd\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Simply add an index to each existing column except the first one which is a target:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def criteo_to_libsvm(row):\n",
" return (\n",
" str(row[0])\n",
" + ' '\n",
" + ' '.join(\n",
" [\n",
" # integer features\n",
" str(i) + ':' + str(row[i])\n",
" for i in range(1, 13 + 1)\n",
" if row[i] is not None\n",
" ] + [\n",
" # string features converted from hex to int\n",
" str(i) + ':' + str(int(row[i], 16))\n",
" for i in range(14, 39 + 1)\n",
" if row[i] is not None\n",
" ]\n",
" )\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Do it for all days:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"convert_chunked_data(criteo_day_template, libsvm_day_template, days, load_criteo_rdd, criteo_to_libsvm)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### LibSVM → Train and test (sampling)\n",
"[_(back to toc)_](#Table-of-contents)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's name samples as their shortened \"engineering\" notation - e.g. 1e5 is 100k etc.:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def sample_name(sample):\n",
" return str(sample)[::-1].replace('000', 'k')[::-1]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Load data, sample a bit more than needed and cut at exact desired number of lines by zipping with index and filtering upto required index:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"oversample = 1.03\n",
"sampled_partitions = 256\n",
"\n",
"\n",
"def sample_and_save(input_path_template, output_path_template, days, samples):\n",
" union = None\n",
" union_count = None\n",
" \n",
" for sample in samples:\n",
" name = sample_name(sample)\n",
" output_path = output_path_template.format(name)\n",
" \n",
" if hdfs_success(output_path):\n",
" logger.info('Sample \"%s\" is already written to \"%s\", skipping.', sample, output_path)\n",
" continue\n",
" \n",
" logger.info('Preparing to write sample to \"%s\".', output_path)\n",
" \n",
" if union is None:\n",
" rdds = map(lambda day: sc.textFile(input_path_template.format(day)), days)\n",
" union = reduce(lambda left, right: left.union(right), rdds)\n",
"\n",
" union_count = union.count()\n",
" logger.info('Total number of lines for days \"%s\" is \"%s\".', days, union_count)\n",
" \n",
" ratio = float(sample) / union_count\n",
" \n",
" sampled_union = (\n",
" union\n",
" .sample(False, min(1.0, oversample * ratio))\n",
" .zipWithIndex()\n",
" .filter(lambda z: z[1] < sample)\n",
" .map(lambda z: z[0])\n",
" )\n",
" \n",
" if hdfs_exists(output_path):\n",
" logger.info('Cleaning \"%s\".', output_path)\n",
" hdfs_delete(output_path, recurse=True)\n",
" \n",
" logger.info('Writing sample \"%s\" to \"%s\".', sample, output_path)\n",
" sampled_union.coalesce(sampled_partitions).saveAsTextFile(output_path)\n",
" \n",
" logger.info('Saved \"%s\" lines to \"%s\".', sc.textFile(output_path).count(), output_path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Sample all LibSVM data:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sample_and_save(libsvm_day_template, libsvm_test_template, days[-1:], test_samples)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sample_and_save(libsvm_day_template, libsvm_train_template, days[:-1], train_samples)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### LibSVM train and test → VW train and test\n",
"[_(back to toc)_](#Table-of-contents)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"LibSVM RDD is a text file:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def load_libsvm_rdd(path):\n",
" return sc.textFile(path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Conversion is trivial - we only have to map target to {-1, 1} and convert categorical features to VW feature names as a whole:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def libsvm_to_vw(line):\n",
" parts = line.split(' ')\n",
" parts[0] = '1 |' if parts[0] == '1' else '-1 |'\n",
" for i in range(1, len(parts)):\n",
" index, _, value = parts[i].partition(':')\n",
" if int(index) >= 14:\n",
" parts[i] = index + '_' + value\n",
" return ' '.join(parts)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Also, data for VW should be well shuffled:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import hashlib\n",
"\n",
"\n",
"def calculate_hash(something):\n",
" m = hashlib.md5()\n",
" m.update(str(something))\n",
" return m.hexdigest()\n",
"\n",
"def random_sort(rdd):\n",
" return (\n",
" rdd\n",
" .zipWithIndex()\n",
" .sortBy(lambda z: calculate_hash(z[1]))\n",
" .map(lambda z: z[0])\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Convert all LibSVM samples:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"convert_chunked_data(libsvm_test_template, vw_test_template, [sample_name(sample) for sample in test_samples], load_libsvm_rdd, libsvm_to_vw, transform_rdd=random_sort)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"convert_chunked_data(libsvm_train_template, vw_train_template, [sample_name(sample) for sample in train_samples], load_libsvm_rdd, libsvm_to_vw, transform_rdd=random_sort)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Spark is no longer needed:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"spark.stop()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Local data\n",
"[_(back to toc)_](#Table-of-contents)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Download all sampled data to local directory:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"ensure_directory_exists(local_data_path)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def count_lines(path):\n",
" with open(path) as f:\n",
" for i, _ in enumerate(f):\n",
" pass\n",
" return i + 1\n",
"\n",
"def download_data(remote_template, local_template, samples):\n",
" for sample in samples:\n",
" name = sample_name(sample)\n",
" remote_path = remote_template.format(name)\n",
" local_path = local_template.format(name)\n",
" if os.path.exists(local_path):\n",
" count = count_lines(local_path)\n",
" if count == sample:\n",
" logger.info('File \"%s\" is already loaded, skipping.', local_path)\n",
" continue\n",
" else:\n",
" logger.info('File \"%s\" already exists but number of lines \"%s\" is wrong (must be \"%s\"), reloading.', local_path, count, sample)\n",
" logger.info('Loading file \"%s\" as local file \"%s\".', remote_path, local_path)\n",
" hdfs_get(remote_path, local_path)\n",
" count = count_lines(local_path)\n",
" logger.info('File loaded to \"%s\", number of lines is \"%s\".', local_path, count)\n",
" assert count == sample, 'File \"{}\" contains wrong number of lines \"{}\" (must be \"{}\").'.format(local_path, count, sample)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"download_data(libsvm_test_template, local_libsvm_test_template, test_samples)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"download_data(libsvm_train_template, local_libsvm_train_template, train_samples)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"download_data(vw_test_template, local_vw_test_template, test_samples)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"download_data(vw_train_template, local_vw_train_template, train_samples)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Local training\n",
"[_(back to toc)_](#Table-of-contents)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Measuring model quality and ML engine technical metrics:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import sys \n",
"from matplotlib import pyplot\n",
"from sklearn.metrics import (\n",
" auc,\n",
" log_loss,\n",
" roc_curve,\n",
")\n",
"\n",
"\n",
"def measure(engine, sample, test_file, time_file, predictions_file):\n",
" \n",
" def get_last_in_line(s):\n",
" return s.rstrip().split( )[-1]\n",
"\n",
" def parse_elapsed_time(s):\n",
" return reduce(lambda a, b: a * 60 + b, map(float, get_last_in_line(s).split(':')))\n",
"\n",
" def parse_max_memory(s):\n",
" return int(get_last_in_line(s)) * 1024\n",
"\n",
" def parse_cpu(s):\n",
" return float(get_last_in_line(s).rstrip('%')) / 100 \n",
"\n",
"\n",
" elapsed = -1\n",
" memory = -1\n",
" cpu = -1\n",
"\n",
" with open(time_file, 'rb') as f:\n",
" for line in f:\n",
" if 'Elapsed (wall clock) time' in line:\n",
" elapsed = parse_elapsed_time(line)\n",
" elif 'Maximum resident set size' in line:\n",
" memory = parse_max_memory(line)\n",
" elif 'Percent of CPU' in line:\n",
" cpu = parse_cpu(line)\n",
"\n",
" with open(test_file, 'rb') as f:\n",
" labels = [line.rstrip().split(' ')[0] == '1' for line in f]\n",
"\n",
" with open(predictions_file, 'rb') as f:\n",
" scores = [float(line.rstrip().split(' ')[0]) for line in f]\n",
"\n",
" fpr, tpr, _ = roc_curve(labels, scores)\n",
" roc_auc = auc(fpr, tpr)\n",
" ll = log_loss(labels, scores)\n",
" \n",
" figure = pyplot.figure(figsize=(6, 6))\n",
" pyplot.plot(fpr, tpr, linewidth=2.0)\n",
" pyplot.plot([0, 1], [0, 1], 'k--')\n",
" pyplot.xlabel('FPR')\n",
" pyplot.ylabel('TPR')\n",
" pyplot.title('{} {} - {:.3f} ROC AUC'.format(engine, sample_name(sample), roc_auc))\n",
" pyplot.show()\n",
"\n",
" return {\n",
" 'Engine': engine,\n",
" 'Train size': sample,\n",
" 'ROC AUC': roc_auc,\n",
" 'Log loss': ll,\n",
" 'Train time': elapsed,\n",
" 'Maximum memory': memory,\n",
" 'CPU load': cpu,\n",
" }"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Settings for VW & XGBoost and how to run them; I use (a little bit patched for correctness sake) GNU Time to measure running time, CPU load and memory consumption; configurations for VW & XGBoost are obtained via Hyperopt:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def get_time_command_and_file(train_file):\n",
" time_file = train_file + '.time'\n",
" return [\n",
" '/usr/local/bin/time',\n",
" '-v',\n",
" '--output=' + time_file,\n",
" ], time_file\n",
"\n",
"def get_vw_commands_and_predictions_file(train_file, test_file):\n",
" model_file = train_file + '.model'\n",
" predictions_file = test_file + '.predictions'\n",
" return [\n",
" 'vw83',\n",
" '--link=logistic',\n",
" '--loss_function=logistic',\n",
" '-b', '29',\n",
" '-l', '0.3',\n",
" '--initial_t', '1',\n",
" '--decay_learning_rate', '0.5',\n",
" '--power_t', '0.5',\n",
" '--l1', '1e-15',\n",
" '--l2', '0',\n",
" '-d', train_file,\n",
" '-f', model_file,\n",
" ], [\n",
" 'vw83',\n",
" '--loss_function=logistic',\n",
" '-t',\n",
" '-i', model_file,\n",
" '-d', test_file,\n",
" '-p', predictions_file,\n",
" ], predictions_file\n",
"\n",
"\n",
"xgboost_conf = [\n",
" 'booster = gbtree',\n",
" 'objective = binary:logistic',\n",
" 'nthread = 24',\n",
" 'eval_metric = logloss',\n",
" 'max_depth = 7',\n",
" 'num_round = 200',\n",
" 'eta = 0.2',\n",
" 'gamma = 0.4',\n",
" 'subsample = 0.8',\n",
" 'colsample_bytree = 0.8',\n",
" 'min_child_weight = 20',\n",
" 'alpha = 3',\n",
" 'lambda = 100',\n",
"]\n",
"\n",
"\n",
"def get_xgboost_commands_and_predictions_file(train_file, test_file, cache=False):\n",
" config_file = os.path.join(local_runtime_path, 'xgb.conf')\n",
" ensure_directory_exists(local_runtime_path)\n",
" with open(config_file, 'wb') as f:\n",
" for line in xgboost_conf:\n",
" print(line, file=f)\n",
" model_file = train_file + '.model'\n",
" predictions_file = test_file + '.predictions'\n",
" if cache:\n",
" train_file = train_file + '#' + train_file + '.cache'\n",
" return [\n",
" 'xgboost',\n",
" config_file,\n",
" 'data=' + train_file,\n",
" 'model_out=' + model_file,\n",
" ], [\n",
" 'xgboost',\n",
" config_file,\n",
" 'task=pred',\n",
" 'test:data=' + test_file,\n",
" 'model_in=' + model_file,\n",
" 'name_pred=' + predictions_file,\n",
" ], predictions_file\n",
"\n",
"def get_xgboost_ooc_commands_and_predictions_file(train_file, test_file):\n",
" return get_xgboost_commands_and_predictions_file(train_file, test_file, cache=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"engines = {\n",
" 'vw': (get_vw_commands_and_predictions_file, local_vw_train_template, local_vw_test_template),\n",
" 'xgb': (get_xgboost_commands_and_predictions_file, local_libsvm_train_template, local_libsvm_test_template),\n",
" 'xgb.ooc': (get_xgboost_ooc_commands_and_predictions_file, local_libsvm_train_template, local_libsvm_test_template),\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Train & test everything:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"import subprocess\n",
"\n",
"\n",
"measurements = []\n",
"\n",
"for sample in train_samples:\n",
" for engine in engines:\n",
" logger.info('Training \"%s\" on \"%s\" lines of data.', engine, sample)\n",
" \n",
" get_commands_and_predictions_file, train_template, test_template = engines[engine]\n",
"\n",
" train_file = train_template.format(sample_name(sample))\n",
" test_file = test_template.format(sample_name(test_samples[0]))\n",
" logger.info('Will train on \"%s\" and test on \"%s\".', train_file, test_file)\n",
"\n",
" command_time, time_file = get_time_command_and_file(train_file)\n",
" command_engine_train, command_engine_test, predictions_file = get_commands_and_predictions_file(train_file, test_file)\n",
"\n",
" logger.info('Performing train.')\n",
" subprocess.call(command_time + command_engine_train)\n",
"\n",
" logger.info('Performing test.')\n",
" subprocess.call(command_engine_test)\n",
"\n",
" logger.info('Measuring results.')\n",
" measurement = measure(engine, sample, test_file, time_file, predictions_file)\n",
" logger.info(measurement)\n",
" measurements.append(measurement)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Load measurements:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pandas\n",
"\n",
"\n",
"measurements_df = pandas.DataFrame(measurements).sort_values(by=['Engine', 'Train size'])\n",
"measurements_df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Plot measurements:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"def extract_data_for_plotting(df, what):\n",
" return reduce(\n",
" lambda left, right: pandas.merge(left, right, how='outer', on='Train size'),\n",
" map(\n",
" lambda name: df[df.Engine == name][['Train size', what]].rename(columns={what: name}),\n",
" df.Engine.unique(),\n",
" ),\n",
" ) \n",
"\n",
"def plot_stuff(df, what, ylabel=None, **kwargs):\n",
" data = extract_data_for_plotting(df, what).set_index('Train size')\n",
" ax = data.plot(marker='o', figsize=(6, 6), title=what, grid=True, linewidth=2.0, **kwargs) # xlim=(1e4, 4e9)\n",
" if ylabel is not None:\n",
" ax.set_ylabel(ylabel)\n",
"\n",
"\n",
"plot_stuff(measurements_df, 'ROC AUC', logx=True)\n",
"plot_stuff(measurements_df, 'Log loss', logx=True)\n",
"plot_stuff(measurements_df, 'Train time', loglog=True, ylabel='s')\n",
"plot_stuff(measurements_df, 'Maximum memory', loglog=True, ylabel='bytes')\n",
"plot_stuff(measurements_df, 'CPU load', logx=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.9"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: notebooks/experiment_spark_lr.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"# Criteo 1 TiB benchmark - Spark.ML logistic regression\n",
"\n",
"Specialization of the experiment notebook for Spark.ML logistic regression."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Table of contents\n",
"\n",
"* [Configuration](#Configuration)\n",
"* [Distributed training](#Distributed-training)\n",
"* [End](#End)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"%load_ext autotime\n",
"%matplotlib inline\n",
"\n",
"from __future__ import print_function"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Configuration\n",
"[_(back to toc)_](#Table-of-contents)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Paths:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"libsvm_data_remote_path = 'criteo/libsvm'\n",
"local_runtime_path = 'criteo/runtime'"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import os\n",
"\n",
"\n",
"libsvm_train_template = os.path.join(libsvm_data_remote_path, 'train', '{}')\n",
"libsvm_test_template = os.path.join(libsvm_data_remote_path, 'test', '{}')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def ensure_directory_exists(path):\n",
" if not os.path.exists(path):\n",
" os.makedirs(path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Samples to take:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"train_samples = [\n",
" 10000, 30000, # tens of thousands\n",
" 100000, 300000, # hundreds of thousands\n",
" 1000000, 3000000, # millions\n",
" 10000000, 30000000, # tens of millions\n",
" 100000000, 300000000, # hundreds of millions\n",
" 1000000000, 3000000000, # billions\n",
"]\n",
"\n",
"test_samples = [1000000]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Spark configuration and initialization:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"total_cores = 256"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"executor_cores = 4\n",
"executor_instances = total_cores / executor_cores\n",
"memory_per_core = 2"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"app_name = 'Criteo experiment - LR on 128 cores'\n",
"\n",
"master = 'yarn'\n",
"\n",
"settings = {\n",
" 'spark.network.timeout': '600',\n",
" \n",
" 'spark.driver.cores': '16',\n",
" 'spark.driver.maxResultSize': '16G',\n",
" 'spark.driver.memory': '32G',\n",
" \n",
" 'spark.executor.cores': str(executor_cores),\n",
" 'spark.executor.instances': str(executor_instances),\n",
" 'spark.executor.memory': str(memory_per_core * executor_cores) + 'G',\n",
" \n",
" 'spark.speculation': 'true',\n",
" \n",
" 'spark.yarn.queue': 'root.HungerGames',\n",
"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": false
},
"outputs": [],
"source": [
"from pyspark.sql import SparkSession\n",
"\n",
"\n",
"builder = SparkSession.builder\n",
"\n",
"builder.appName(app_name)\n",
"builder.master(master)\n",
"for k, v in settings.items():\n",
" builder.config(k, v)\n",
"\n",
"spark = builder.getOrCreate()\n",
"sc = spark.sparkContext\n",
"\n",
"sc.setLogLevel('ERROR')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Logging:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import sys\n",
"import logging\n",
"reload(logging)\n",
"\n",
"\n",
"handler = logging.StreamHandler(stream=sys.stdout)\n",
"formatter = logging.Formatter('[%(asctime)s] %(message)s')\n",
"handler.setFormatter(formatter)\n",
"\n",
"ensure_directory_exists(local_runtime_path)\n",
"file_handler = logging.FileHandler(filename=os.path.join(local_runtime_path, 'mylog.log'), mode='a')\n",
"file_handler.setFormatter(formatter)\n",
"\n",
"logger = logging.getLogger()\n",
"logger.addHandler(handler)\n",
"logger.addHandler(file_handler)\n",
"logger.setLevel(logging.DEBUG)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"logger.info('Spark version: %s.', spark.version)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Plot measurements:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": false
},
"outputs": [],
"source": [
"import pandas\n",
"\n",
"\n",
"def extract_data_for_plotting(df, what):\n",
" return reduce(\n",
" lambda left, right: pandas.merge(left, right, how='outer', on='Train size'),\n",
" map(\n",
" lambda name: df[df.Engine == name][['Train size', what]].rename(columns={what: name}),\n",
" df.Engine.unique(),\n",
" ),\n",
" ) \n",
"\n",
"def plot_stuff(df, what, ylabel=None, **kwargs):\n",
" data = extract_data_for_plotting(df, what).set_index('Train size')\n",
" ax = data.plot(marker='o', figsize=(6, 6), title=what, grid=True, linewidth=2.0, **kwargs) # xlim=(1e4, 4e9)\n",
" if ylabel is not None:\n",
" ax.set_ylabel(ylabel)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's name samples as their shortened \"engineering\" notation - 1e5 is 100k etc.:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def sample_name(sample):\n",
" return str(sample)[::-1].replace('000', 'k')[::-1]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Distributed training\n",
"[_(back to toc)_](#Table-of-contents)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Loading of LibSVM data as Spark.ML dataset:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import hashlib\n",
"import math\n",
"import struct\n",
"from pyspark.ml.linalg import SparseVector\n",
"\n",
"\n",
"total_features = 100000\n",
"\n",
"\n",
"def hash_fun(x):\n",
" return int(struct.unpack('L', hashlib.md5(x).digest()[:8])[0] % total_features)\n",
"\n",
"def parse_kv_pair(kv):\n",
" k, _, v = kv.partition(':')\n",
" return int(k), int(v)\n",
"\n",
"def parse_libsvm_line(line):\n",
" parts = line.split(' ')\n",
" \n",
" label = int(parts[0])\n",
" pairs = map(parse_kv_pair, parts[1:])\n",
" \n",
" for i in range(len(pairs)):\n",
" k, v = pairs[i]\n",
" if k < 14:\n",
" if v > 2:\n",
" pairs[i] = (k, int(math.log(v) ** 2))\n",
" else:\n",
" pairs[i] = (k, '{:08x}'.format(v))\n",
" \n",
" indices = sorted({hash_fun('{}_{}'.format(k, v)) for k, v in pairs})\n",
" values = [1.0] * len(indices)\n",
" \n",
" return (label, SparseVector(total_features, indices, values))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"task_splitting = 1 # tasks per core\n",
"\n",
"def load_ml_data(template, sample):\n",
" path = template.format(sample_name(sample))\n",
" return sc.textFile(path).map(parse_libsvm_line).toDF(['label', 'features']).repartition(executor_cores * executor_instances * task_splitting)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Evaluating a model:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from matplotlib import pyplot\n",
"from sklearn.metrics import auc, log_loss, roc_curve\n",
"\n",
"\n",
"def calculate_roc(predictions):\n",
" labels, scores = zip(*predictions.rdd.map(lambda row: (row.label, row.probability[1])).collect())\n",
" fpr, tpr, _ = roc_curve(labels, scores)\n",
" roc_auc = auc(fpr, tpr)\n",
" ll = log_loss(labels, scores)\n",
" return fpr, tpr, roc_auc, ll\n",
"\n",
"def evaluate_model(name, model, test, train=None):\n",
" metrics = dict()\n",
" \n",
" figure = pyplot.figure(figsize=(6, 6))\n",
" ax = figure.gca()\n",
" ax.set_title('ROC - ' + name)\n",
" \n",
" if train is not None:\n",
" train_predictions = model.transform(train)\n",
" train_fpr, train_tpr, train_roc_auc, train_log_loss = calculate_roc(train_predictions)\n",
" \n",
" metrics['train_roc_auc'] = train_roc_auc\n",
" metrics['train_log_loss'] = train_log_loss\n",
" \n",
" ax.plot(train_fpr, train_tpr, linewidth=2.0, label='train (auc = {:.3f})'.format(train_roc_auc))\n",
" \n",
" test_predictions = model.transform(test)\n",
" test_fpr, test_tpr, test_roc_auc, test_log_loss = calculate_roc(test_predictions)\n",
" \n",
" metrics['test_roc_auc'] = test_roc_auc\n",
" metrics['test_log_loss'] = test_log_loss\n",
" \n",
" ax.plot(test_fpr, test_tpr, linewidth=2.0, label='test (auc = {:.3f})'.format(test_roc_auc))\n",
" \n",
" ax.plot([0.0, 1.0], [0.0, 1.0], linestyle='--', c='gray')\n",
" ax.legend()\n",
" pyplot.show()\n",
" \n",
" return metrics"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Models to work on:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from pyspark.ml.classification import (\n",
" LogisticRegression,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"classifiers = {\n",
" 'lr': LogisticRegression(regParam=0.03),\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Monkey-patch RDDs and DataFrames for context persistence:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import pyspark\n",
"\n",
"\n",
"def enter_method(self):\n",
" self.persist()\n",
"\n",
"def exit_method(self,exc_type, exc, traceback):\n",
" self.unpersist()\n",
"\n",
"\n",
"pyspark.sql.dataframe.DataFrame.__enter__ = enter_method\n",
"pyspark.sql.dataframe.DataFrame.__exit__ = exit_method"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Do distributed training:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": false
},
"outputs": [],
"source": [
"import time\n",
"\n",
"from pyspark.ml.evaluation import BinaryClassificationEvaluator\n",
"\n",
"\n",
"test_sample = test_samples[-1]\n",
"\n",
"evaluator = BinaryClassificationEvaluator(labelCol='label', rawPredictionCol='probability', metricName='areaUnderROC')\n",
"\n",
"new_quality_data = []\n",
"new_data_rows = []\n",
"\n",
"test_dfs = dict()\n",
"\n",
"logger.info('Loading \"%s\" test samples.', test_sample)\n",
"test_df = load_ml_data(libsvm_test_template, test_sample)\n",
"with test_df:\n",
" logger.info('Loaded \"%s\" lines.', test_df.count())\n",
"\n",
" for train_sample in train_samples:\n",
"\n",
" logger.info('Working on \"%s\" train sample.', train_sample)\n",
"\n",
" for classifier_name, classifier in classifiers.items():\n",
"\n",
" logger.info('Training a model \"%s\" on sample \"%s\".', classifier_name, train_sample)\n",
"\n",
" logger.info('Loading \"%s\" train samples.', train_sample)\n",
" train_df = load_ml_data(libsvm_train_template, train_sample)\n",
" with train_df:\n",
" logger.info('Loaded \"%s\" lines.', train_df.count())\n",
" \n",
" logger.info('Training a model \"%s\" on sample \"%s\".', classifier_name, train_sample)\n",
"\n",
" start = time.time()\n",
" model = classifier.fit(train_df)\n",
" duration = time.time() - start\n",
"\n",
" logger.info('Training a model \"%s\" on sample \"%s\" took \"%g\" seconds.', classifier_name, train_sample, duration)\n",
"\n",
" logger.info('Evaluating the model \"%s\" trained on sample \"%s\".', classifier_name, train_sample)\n",
" metrics = evaluate_model(classifier_name + ' - ' + sample_name(train_sample), model, test_df, train=(train_df if train_sample <= 1000000 else None))\n",
"\n",
" test_predictions = model.transform(test_df)\n",
" ml_metric_value = evaluator.evaluate(test_predictions)\n",
"\n",
" logger.info(\n",
" 'For the model \"%s\" trained on sample \"%s\" metrics are: \"%s\"; ROC AUC calculated by Spark is \"%s\".',\n",
" classifier_name,\n",
" train_sample,\n",
" metrics,\n",
" ml_metric_value,\n",
" )\n",
"\n",
" data_row = {\n",
" 'Train size': train_sample,\n",
" 'ROC AUC': metrics['test_roc_auc'],\n",
" 'Log loss': metrics['test_log_loss'],\n",
" 'Duration': duration,\n",
" 'Engine': classifier_name,\n",
" }\n",
" new_quality_data.append(data_row)\n",
" data_row_string = '\\t'.join(str(data_row[field]) for field in ['Engine', 'Train size', 'ROC AUC', 'Log loss', 'Duration'])\n",
" new_data_rows.append(data_row_string)\n",
" logger.info('Data row: \"%s\".', data_row_string)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": false
},
"outputs": [],
"source": [
"measurements_df = pandas.DataFrame(new_quality_data).sort_values(by=['Train size'])\n",
"plot_stuff(measurements_df, 'ROC AUC', logx=True)\n",
"plot_stuff(measurements_df, 'Log loss', logx=True, ylim=(0.13, 0.18))\n",
"plot_stuff(measurements_df, 'Duration', loglog=True, ylabel='s')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"for row in new_data_rows:\n",
" print(row)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## End\n",
"[_(back to toc)_](#Table-of-contents)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Work done, stop Spark:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"spark.stop()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.1"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: notebooks/experiment_spark_rf.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Criteo 1 TiB benchmark - Spark.ML random forest\n",
"\n",
"Specialization of the experimental notebook for Spark.ML random forest."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Table of contents\n",
"\n",
"* [Configuration](#Configuration)\n",
"* [Distributed training](#Distributed-training)\n",
"* [End](#End)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"%load_ext autotime\n",
"%matplotlib inline\n",
"\n",
"from __future__ import print_function"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Configuration\n",
"[_(back to toc)_](#Table-of-contents)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Paths:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"libsvm_data_remote_path = 'criteo/libsvm'\n",
"local_runtime_path = 'criteo/runtime'"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import os\n",
"\n",
"\n",
"libsvm_train_template = os.path.join(libsvm_data_remote_path, 'train', '{}')\n",
"libsvm_test_template = os.path.join(libsvm_data_remote_path, 'test', '{}')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def ensure_directory_exists(path):\n",
" if not os.path.exists(path):\n",
" os.makedirs(path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Samples to take:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"train_samples = [\n",
" 10000, 30000, # tens of thousands\n",
" 100000, 300000, # hundreds of thousands\n",
" 1000000, 3000000, # millions\n",
" 10000000, 30000000, # tens of millions\n",
" 100000000, 300000000, # hundreds of millions\n",
" 1000000000, 3000000000, # billions\n",
"]\n",
"\n",
"test_samples = [1000000]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Spark configuration and initialization:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"executor_instances = 64\n",
"executor_cores = 4\n",
"memory_per_core = 4"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"app_name = 'Criteo experiment'\n",
"\n",
"master = 'yarn'\n",
"\n",
"settings = {\n",
" 'spark.network.timeout': '600',\n",
" \n",
" 'spark.driver.cores': '16',\n",
" 'spark.driver.maxResultSize': '16G',\n",
" 'spark.driver.memory': '32G',\n",
" \n",
" 'spark.executor.cores': str(executor_cores),\n",
" 'spark.executor.instances': str(executor_instances),\n",
" 'spark.executor.memory': str(memory_per_core * executor_cores) + 'G',\n",
" \n",
" 'spark.speculation': 'true',\n",
" \n",
" 'spark.yarn.queue': 'root.HungerGames',\n",
"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": false
},
"outputs": [],
"source": [
"from pyspark.sql import SparkSession\n",
"\n",
"\n",
"builder = SparkSession.builder\n",
"\n",
"builder.appName(app_name)\n",
"builder.master(master)\n",
"for k, v in settings.items():\n",
" builder.config(k, v)\n",
"\n",
"spark = builder.getOrCreate()\n",
"sc = spark.sparkContext\n",
"\n",
"sc.setLogLevel('ERROR')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Logging:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import sys\n",
"import logging\n",
"reload(logging)\n",
"\n",
"\n",
"handler = logging.StreamHandler(stream=sys.stdout)\n",
"formatter = logging.Formatter('[%(asctime)s] %(message)s')\n",
"handler.setFormatter(formatter)\n",
"\n",
"ensure_directory_exists(local_runtime_path)\n",
"file_handler = logging.FileHandler(filename=os.path.join(local_runtime_path, 'mylog.log'), mode='a')\n",
"file_handler.setFormatter(formatter)\n",
"\n",
"logger = logging.getLogger()\n",
"logger.addHandler(handler)\n",
"logger.addHandler(file_handler)\n",
"logger.setLevel(logging.DEBUG)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"logger.info('Spark version: %s.', spark.version)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Plot measurements:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": false
},
"outputs": [],
"source": [
"import pandas\n",
"\n",
"\n",
"def extract_data_for_plotting(df, what):\n",
" return reduce(\n",
" lambda left, right: pandas.merge(left, right, how='outer', on='Train size'),\n",
" map(\n",
" lambda name: df[df.Engine == name][['Train size', what]].rename(columns={what: name}),\n",
" df.Engine.unique(),\n",
" ),\n",
" ) \n",
"\n",
"def plot_stuff(df, what, ylabel=None, **kwargs):\n",
" data = extract_data_for_plotting(df, what).set_index('Train size')\n",
" ax = data.plot(marker='o', figsize=(6, 6), title=what, grid=True, linewidth=2.0, **kwargs) # xlim=(1e4, 4e9)\n",
" if ylabel is not None:\n",
" ax.set_ylabel(ylabel)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's name samples as their shortened \"engineering\" notation - 1e5 is 100k etc.:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def sample_name(sample):\n",
" return str(sample)[::-1].replace('000', 'k')[::-1]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Distributed training\n",
"[_(back to toc)_](#Table-of-contents)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Loading of LibSVM data as Spark.ML dataset:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from pyspark.ml.linalg import SparseVector\n",
"\n",
"\n",
"def parse_libsvm_line_for_rf(line):\n",
" parts = line.split(' ')\n",
" label = int(parts[0])\n",
" indices, values = zip(*map(lambda s: s.split(':'), parts[1:]))\n",
" return (label, SparseVector(40, map(int, indices), map(int, values)))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"task_splitting = 1 # tasks per core\n",
"\n",
"def load_ml_data_for_rf(template, sample):\n",
" path = template.format(sample_name(sample))\n",
" return sc.textFile(path).map(parse_libsvm_line_for_rf).toDF(['label', 'features']).repartition(executor_cores * executor_instances * task_splitting)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Evaluating a model:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from matplotlib import pyplot\n",
"from sklearn.metrics import auc, log_loss, roc_curve\n",
"\n",
"\n",
"def calculate_roc(predictions):\n",
" labels, scores = zip(*predictions.rdd.map(lambda row: (row.label, row.probability[1])).collect())\n",
" fpr, tpr, _ = roc_curve(labels, scores)\n",
" roc_auc = auc(fpr, tpr)\n",
" ll = log_loss(labels, scores)\n",
" return fpr, tpr, roc_auc, ll\n",
"\n",
"def evaluate_model(name, model, test, train=None):\n",
" metrics = dict()\n",
" \n",
" figure = pyplot.figure(figsize=(6, 6))\n",
" ax = figure.gca()\n",
" ax.set_title('ROC - ' + name)\n",
" \n",
" if train is not None:\n",
" train_predictions = model.transform(train)\n",
" train_fpr, train_tpr, train_roc_auc, train_log_loss = calculate_roc(train_predictions)\n",
" \n",
" metrics['train_roc_auc'] = train_roc_auc\n",
" metrics['train_log_loss'] = train_log_loss\n",
" \n",
" ax.plot(train_fpr, train_tpr, linewidth=2.0, label='train (auc = {:.3f})'.format(train_roc_auc))\n",
" \n",
" test_predictions = model.transform(test)\n",
" test_fpr, test_tpr, test_roc_auc, test_log_loss = calculate_roc(test_predictions)\n",
" \n",
" metrics['test_roc_auc'] = test_roc_auc\n",
" metrics['test_log_loss'] = test_log_loss\n",
" \n",
" ax.plot(test_fpr, test_tpr, linewidth=2.0, label='test (auc = {:.3f})'.format(test_roc_auc))\n",
" \n",
" ax.plot([0.0, 1.0], [0.0, 1.0], linestyle='--', c='gray')\n",
" ax.legend()\n",
" pyplot.show()\n",
" \n",
" return metrics"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Models to work on:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from pyspark.ml.classification import (\n",
" RandomForestClassifier, \n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"classifiers = {\n",
" 'rf': RandomForestClassifier(featureSubsetStrategy='sqrt', impurity='entropy', minInstancesPerNode=3, maxBins=64, maxDepth=10, numTrees=160),\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Monkey-patch RDDs and DataFrames for context persistence:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import pyspark\n",
"\n",
"\n",
"def enter_method(self):\n",
" self.persist()\n",
"\n",
"def exit_method(self,exc_type, exc, traceback):\n",
" self.unpersist()\n",
"\n",
"\n",
"pyspark.sql.dataframe.DataFrame.__enter__ = enter_method\n",
"pyspark.sql.dataframe.DataFrame.__exit__ = exit_method"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Do distributed training:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": false
},
"outputs": [],
"source": [
"import time\n",
"\n",
"from pyspark.ml.evaluation import BinaryClassificationEvaluator\n",
"\n",
"\n",
"test_sample = test_samples[-1]\n",
"\n",
"evaluator = BinaryClassificationEvaluator(labelCol='label', rawPredictionCol='probability', metricName='areaUnderROC')\n",
"\n",
"new_quality_data = []\n",
"new_data_rows = []\n",
"\n",
"logger.info('Loading \"%s\" test samples for rf.', test_sample)\n",
"test_df = load_ml_data_for_rf(libsvm_test_template, test_sample)\n",
"with test_df:\n",
" logger.info('Loaded \"%s\" lines.', test_df.count())\n",
"\n",
" for train_sample in train_samples:\n",
"\n",
" logger.info('Working on \"%s\" train sample.', train_sample)\n",
"\n",
" logger.info('Loading \"%s\" train samples for rf.', train_sample)\n",
" train_df = load_ml_data_for_rf(libsvm_train_template, train_sample)\n",
" with train_df:\n",
" logger.info('Loaded \"%s\" lines.', train_df.count())\n",
"\n",
" for classifier_name, classifier in classifiers.items():\n",
"\n",
" logger.info('Training a model \"%s\" on sample \"%s\".', classifier_name, train_sample)\n",
"\n",
" start = time.time()\n",
" model = classifier.fit(train_df)\n",
" duration = time.time() - start\n",
"\n",
" logger.info('Training a model \"%s\" on sample \"%s\" took \"%g\" seconds.', classifier_name, train_sample, duration)\n",
"\n",
" logger.info('Evaluating the model \"%s\" trained on sample \"%s\".', classifier_name, train_sample)\n",
" metrics = evaluate_model(classifier_name + ' - ' + sample_name(train_sample), model, test_df, train=(train_df if train_sample <= 1000000 else None))\n",
"\n",
" test_predictions = model.transform(test_df)\n",
" ml_metric_value = evaluator.evaluate(test_predictions)\n",
"\n",
" logger.info(\n",
" 'For the model \"%s\" trained on sample \"%s\" metrics are: \"%s\"; ROC AUC calculated by Spark is \"%s\".',\n",
" classifier_name,\n",
" train_sample,\n",
" metrics,\n",
" ml_metric_value,\n",
" )\n",
"\n",
" data_row = {\n",
" 'Train size': train_sample,\n",
" 'ROC AUC': metrics['test_roc_auc'],\n",
" 'Log loss': metrics['test_log_loss'],\n",
" 'Duration': duration,\n",
" 'Engine': classifier_name,\n",
" }\n",
" new_quality_data.append(data_row)\n",
" data_row_string = '\\t'.join(str(data_row[field]) for field in ['Engine', 'Train size', 'ROC AUC', 'Log loss', 'Duration'])\n",
" new_data_rows.append(data_row_string)\n",
" logger.info('Data row: \"%s\".', data_row_string)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Plot metrics:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": false
},
"outputs": [],
"source": [
"measurements_df = pandas.DataFrame(new_quality_data).sort_values(by=['Train size'])\n",
"plot_stuff(measurements_df, 'ROC AUC', logx=True)\n",
"plot_stuff(measurements_df, 'Log loss', logx=True, ylim=(0.135, 0.145))\n",
"plot_stuff(measurements_df, 'Duration', loglog=True, ylabel='s')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"for row in new_data_rows:\n",
" print(row)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## End\n",
"[_(back to toc)_](#Table-of-contents)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Work done, stop Spark:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"spark.stop()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.1"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: results/metrics.cluster.tsv
================================================
"Engine" "Train size" "ROC AUC" "Log loss" "Train time"
"lr" 10000 0.577664202812 0.191408441626 50.9623498917
"lr" 30000 0.593790859746 0.181353099861 87.1837468147
"lr" 100000 0.612247160316 0.169781474994 148.152549982
"lr" 300000 0.644818188231 0.155259096662 138.703317881
"lr" 1000000 0.684101056031 0.142522071064 154.500109911
"lr" 3000000 0.720519424179 0.135748217546 140.207423925
"lr" 10000000 0.742150224051 0.133061793307 175.111939907
"lr" 30000000 0.750851634783 0.132163191494 205.059295177
"lr" 100000000 0.754105416714 0.131909898252 429.173339128
"lr" 300000000 0.755291615978 0.131809908334 655.645493984
"lr" 1000000000 0.755555957222 0.131795697268 5133.28028798
"lr" 3000000000 0.755575083657 0.131792462522 5726.66465712
"rf" 10000 0.666209405818 0.141072504942 162.25733614
"rf" 30000 0.686325476035 0.139292662004 123.259371996
"rf" 100000 0.698154160624 0.138366116883 67.557528019
"rf" 300000 0.704939649921 0.137753153698 78.4612021446
"rf" 1000000 0.7072363872 0.137494568389 105.36420989
"rf" 3000000 0.707892500228 0.13742745787 154.24793601
"rf" 10000000 0.708613003835 0.13734362384 518.322438955
"rf" 30000000 0.708447321414 0.137352474673 665.707078934
"rf" 100000000 0.708270614391 0.137371388594 1821.9798851
"rf" 300000000 0.708382095174 0.137347952598 3362.20391607
================================================
FILE: results/metrics.lr_hash_size.tsv
================================================
"Engine" "Train size" "ROC AUC" "Log loss" "Train time"
"lr, 100k hashes" 10000 0.577664202812 0.191408441626 50.9623498917
"lr, 100k hashes" 30000 0.593790859746 0.181353099861 87.1837468147
"lr, 100k hashes" 100000 0.612247160316 0.169781474994 148.152549982
"lr, 100k hashes" 300000 0.644818188231 0.155259096662 138.703317881
"lr, 100k hashes" 1000000 0.684101056031 0.142522071064 154.500109911
"lr, 100k hashes" 3000000 0.720519424179 0.135748217546 140.207423925
"lr, 100k hashes" 10000000 0.742150224051 0.133061793307 175.111939907
"lr, 100k hashes" 30000000 0.750851634783 0.132163191494 205.059295177
"lr, 100k hashes" 100000000 0.754105416714 0.131909898252 429.173339128
"lr, 30k hashes" 10000 0.575027699783 0.188237385947 34.5243330002
"lr, 30k hashes" 30000 0.594786944726 0.174772368249 47.6537959576
"lr, 30k hashes" 100000 0.625299136967 0.15797820297 94.8742370605
"lr, 30k hashes" 300000 0.669467206714 0.144246165797 106.976841927
"lr, 30k hashes" 1000000 0.71062957893 0.136591896049 106.445657969
"lr, 30k hashes" 3000000 0.732331332028 0.134246693916 117.792547941
"lr, 30k hashes" 10000000 0.742112278546 0.133347719357 139.843713999
"lr, 30k hashes" 30000000 0.744923892422 0.133133669898 171.719541073
"lr, 30k hashes" 100000000 0.7463477576 0.133017218565 478.496304035
================================================
FILE: results/metrics.old.tsv
================================================
"Engine" "Train size" "ROC AUC" "Log loss" "Train time"
"vw" 10000 0.609 0.1501 1
"vw" 30000 0.632 0.1492 2
"vw" 100000 0.653 0.1461 3
"vw" 300000 0.679 0.1429 4
"vw" 1000000 0.692 0.1413 5
"vw" 3000000 0.707 0.1387 6
"vw" 10000000 0.72 0.1369 7
"xgb" 10000 0.65 0.1472 8
"xgb" 30000 0.67 0.1451 9
"xgb" 100000 0.683 0.1442 10
"xgb" 300000 0.694 0.14355 11
"xgb" 1000000 0.695 0.14335 12
"xgb" 3000000 0.696 0.14325 13
"xgb" 10000000 0.6965 0.14323 14
"xgb.ooc" 10000 0.642 0.1476 15
"xgb.ooc" 30000 0.672 0.1446 16
"xgb.ooc" 100000 0.685 0.144 17
"xgb.ooc" 300000 0.694 0.1435 18
"xgb.ooc" 1000000 0.695 0.1433 19
"xgb.ooc" 3000000 0.696 0.1433 20
"xgb.ooc" 10000000 0.6964 0.143231 21
================================================
FILE: results/metrics.selection.tsv
================================================
"Engine" "Train size" "ROC AUC" "Log loss" "Train time"
lr 10000 0.578177884844 0.178971884565 29.5085098743
rf 10000 0.621504103494 0.144342437741 76.4821279049
tree 10000 0.600422662988 0.357273201261 11.1679940224
bayes 10000 0.509993205115 0.818047776891 1.48214101791
lr 30000 0.606712654428 0.16236564437 35.3462071419
rf 30000 0.643115035038 0.142662388067 92.7048139572
tree 30000 0.617018579741 0.165522218545 11.9768800735
bayes 30000 0.565664291131 0.473385707556 0.820657014847
lr 100000 0.652313535925 0.145698453913 40.4066979885
rf 100000 0.663411563927 0.14115005143 144.407087088
tree 100000 0.638114905876 0.149601338341 12.5886089802
bayes 100000 0.637296917738 0.373915545595 0.909769058228
lr 300000 0.694017701134 0.138406933493 41.6715488434
rf 300000 0.674058380306 0.140104620859 244.973959923
tree 300000 0.641005555466 0.142781786127 16.7563710213
bayes 300000 0.672807318246 0.418186728992 1.29778695107
lr 1000000 0.719469466249 0.135623586024 46.4208109379
rf 1000000 0.677311170308 0.139673520936 586.515081882
tree 1000000 0.64388298082 0.141953441838 29.7467141151
bayes 1000000 0.690009509821 0.437564216119 1.74085712433
lr 3000000 0.729501417959 0.134854378505 52.4596869946
rf 3000000 0.680087853763 0.13939451653 1414.78307986
tree 3000000 0.642813675941 0.141866601496 65.7208981514
bayes 3000000 0.697140532878 0.450650398396 2.11705684662
================================================
FILE: results/metrics.tsv
================================================
"Engine" "Train size" "ROC AUC" "Log loss" "Train time"
"vw" 10000 0.617455161515 0.144524706901 15.17
"vw" 30000 0.647287001387 0.142233353612 13.9
"vw" 100000 0.683247769366 0.139338887152 16.62
"vw" 300000 0.70142316924 0.13745122263 17.75
"vw" 1000000 0.716287545509 0.13588863014 23.83
"vw" 3000000 0.727931938776 0.134788994538 40.1
"vw" 10000000 0.739548558371 0.133521239905 91.08
"vw" 30000000 0.746322264127 0.132553436306 242.17
"vw" 100000000 0.752882908682 0.131795464547 718.71
"vw" 300000000 0.754807741333 0.131569260518 1989.99
"vw" 1000000000 0.757005674498 0.131394975718 6431.0
"vw" 3000000000 0.756123547995 0.132255694498 17798.0
"xgb" 10000 0.663509743693 0.142349739278 1.56
"xgb" 30000 0.682316321881 0.140907061928 4.58
"xgb" 100000 0.696964467698 0.139122124164 14.86
"xgb" 300000 0.714743656593 0.136567639058 45.52
"xgb" 1000000 0.733393270406 0.134198458139 179.49
"xgb" 3000000 0.744253560697 0.13280500935 610.78
"xgb" 10000000 0.750706783469 0.131929939622 2541.02
"xgb" 30000000 0.753289508009 0.131529827949 9065
"xgb" 100000000 0.755224491108 0.131270032458 37105
"xgb.ooc" 10000 0.663936721734 0.142204298072 6.52
"xgb.ooc" 30000 0.684182538527 0.140580030251 24.2
"xgb.ooc" 100000 0.697765852364 0.138864711732 87.48
"xgb.ooc" 300000 0.714633362901 0.136556600616 264.64
"xgb.ooc" 1000000 0.731054714952 0.134485312106 944.91
"xgb.ooc" 3000000 0.742823599115 0.132987550283 2517.86
"xgb.ooc" 10000000 0.750953801983 0.131909674184 5874
"xgb.ooc" 30000000 0.753881106383 0.131430669756 14520
"xgb.ooc" 100000000 0.755746234321 0.131237777868 44875
"lr" 10000 0.577664202812 0.191408441626 50.9623498917
"lr" 30000 0.593790859746 0.181353099861 87.1837468147
"lr" 100000 0.612247160316 0.169781474994 148.152549982
"lr" 300000 0.644818188231 0.155259096662 138.703317881
"lr" 1000000 0.684101056031 0.142522071064 154.500109911
"lr" 3000000 0.720519424179 0.135748217546 140.207423925
"lr" 10000000 0.742150224051 0.133061793307 175.111939907
"lr" 30000000 0.750851634783 0.132163191494 205.059295177
"lr" 100000000 0.754105416714 0.131909898252 429.173339128
"lr" 300000000 0.755291615978 0.131809908334 655.645493984
"lr" 1000000000 0.755555957222 0.131795697268 5133.28028798
"lr" 3000000000 0.755575083657 0.131792462522 5726.66465712
"rf" 10000 0.666209405818 0.141072504942 162.25733614
"rf" 30000 0.686325476035 0.139292662004 123.259371996
"rf" 100000 0.698154160624 0.138366116883 67.557528019
"rf" 300000 0.704939649921 0.137753153698 78.4612021446
"rf" 1000000 0.7072363872 0.137494568389 105.36420989
"rf" 3000000 0.707892500228 0.13742745787 154.24793601
"rf" 10000000 0.708613003835 0.13734362384 518.322438955
"rf" 30000000 0.708447321414 0.137352474673 665.707078934
"rf" 100000000 0.708270614391 0.137371388594 1821.9798851
"rf" 300000000 0.708382095174 0.137347952598 3362.20391607
================================================
FILE: results/vw_xgb.tsv
================================================
"Engine" "Train size" "ROC AUC" "Log loss" "Train time" "Maximum memory" "CPU load"
"vw" 10000 0.617455161515 0.144524706901 15.17 8615268352 1.0
"vw" 30000 0.647287001387 0.142233353612 13.9 8615268352 0.99
"vw" 100000 0.683247769366 0.139338887152 16.62 8615727104 1.01
"vw" 300000 0.70142316924 0.13745122263 17.75 8616218624 1.06
"vw" 1000000 0.716287545509 0.13588863014 23.83 8618024960 1.2
"vw" 3000000 0.727931938776 0.134788994538 40.1 8615903232 1.34
"vw" 10000000 0.739548558371 0.133521239905 91.08 8617746432 1.48
"vw" 30000000 0.746322264127 0.132553436306 242.17 8616017920 1.57
"vw" 100000000 0.752882908682 0.131795464547 718.71 8617304064 1.62
"vw" 300000000 0.754807741333 0.131569260518 1989.99 8616005632 1.67
"vw" 1000000000 0.757005674498 0.131394975718 6431.0 8617570304 1.71
"vw" 3000000000 0.756123547995 0.132255694498 17798.0 8617238528 1.78
"xgb" 10000 0.663509743693 0.142349739278 1.56 18403328 8.4
"xgb" 30000 0.682316321881 0.140907061928 4.58 33456128 10.63
"xgb" 100000 0.696964467698 0.139122124164 14.86 71958528 12.06
"xgb" 300000 0.714743656593 0.136567639058 45.52 193576960 13.07
"xgb" 1000000 0.733393270406 0.134198458139 179.49 617226240 13.2
"xgb" 3000000 0.744253560697 0.13280500935 610.78 1828356096 14.27
"xgb" 10000000 0.750706783469 0.131929939622 2541.02 6056378368 14.12
"xgb" 30000000 0.753289508009 0.131529827949 9065 18139652096 13.51
"xgb" 100000000 0.755224491108 0.131270032458 37105 60420395008 10.78
"xgb.ooc" 10000 0.663936721734 0.142204298072 6.52 31752192 3.71
"xgb.ooc" 30000 0.684182538527 0.140580030251 24.2 48721920 3.68
"xgb.ooc" 100000 0.697765852364 0.138864711732 87.48 109694976 3.72
"xgb.ooc" 300000 0.714633362901 0.136556600616 264.64 279904256 4.09
"xgb.ooc" 1000000 0.731054714952 0.134485312106 944.91 852369408 4.98
"xgb.ooc" 3000000 0.742823599115 0.132987550283 2517.86 1456574464 5.79
"xgb.ooc" 10000000 0.750953801983 0.131909674184 5874 1921560576 8.2
"xgb.ooc" 30000000 0.753881106383 0.131430669756 14520 2562379776 9.95
"xgb.ooc" 100000000 0.755746234321 0.131237777868 44875 4807467008 8.79
================================================
FILE: scripts/conversion/criteoToLibsvm.scala
================================================
def rowToLibsvm(row: org.apache.spark.sql.Row): String = {
0 until row.length flatMap {
case 0 => Some(row(0).toString)
case i if row(i) == null => None
case i => Some(i.toString + ':' + (if (i < 14) row(i) else java.lang.Long.parseLong(row(i).toString, 16)).toString)
} mkString " "
}
def readDataFrame(path: String): org.apache.spark.sql.DataFrame = {
spark.read.option("header", "false").option("inferSchema", "true").option("delimiter", "\t").csv(path)
}
def writeDataFrame(df: org.apache.spark.sql.DataFrame, path: String): Unit = {
df.rdd.map(rowToLibsvm).saveAsTextFile(path)
}
def processDay(day: Int): Unit = {
println(s"Processing of the day $day started")
val inputPath = s"criteo_1tb/plain/day_$day"
println(s"Loading data from $inputPath")
val df = readDataFrame(inputPath)
val outputPath = s"criteo_1tb/libsvm/day_$day.libsvm"
println(s"Saving data to $outputPath")
writeDataFrame(df, outputPath)
println(s"Processing of the day $day finished")
}
println("Do '0 to 23 foreach processDay' to convert all data to LibSVM format.")
================================================
FILE: scripts/conversion/libsvmToVw.scala
================================================
import java.security.MessageDigest
import org.apache.hadoop.fs.{FileSystem, Path => HadoopPath}
import org.apache.spark.rdd.RDD
def convertLine(row: String): String = {
val elements = row.split(' ')
val target = elements.head.toInt * 2 - 1
(target.toString + " |") +: elements.tail.map(e => {
val es = e.split(':')
val index = es(0).toInt
if (index < 14) e
else es.mkString("_")
}) mkString " "
}
def md5[T](s: T) = {
MessageDigest
.getInstance("MD5")
.digest(s.toString.getBytes)
.map("%02x".format(_))
.mkString
}
def convertFile(srcPath: String, dstPath: String) = {
sc
.textFile(srcPath)
.map(convertLine)
.zipWithIndex
.sortBy(z => md5(z._2))
.map(_._1)
.saveAsTextFile(dstPath)
}
val names = List("test", "train")
def powTenAndTriple(n: Int): List[Long] = { val v = scala.math.pow(10, n).longValue; List(v, 3 * v) }
val nums = (4 to 9 flatMap powTenAndTriple).toList
def numToString(num: Long): String = num.toString.reverse.replaceAll("000", "k").reverse
val fs = FileSystem.get(sc.hadoopConfiguration)
def fileExists(path: String): Boolean = fs.exists(new HadoopPath(path))
def removeFile(path: String): Unit = fs.delete(new HadoopPath(path))
def doDataConversion = {
for {
num <- nums
name <- names
numName = numToString(num)
srcPath = s"criteo/libsvm/$name/$numName"
dstPath = s"criteo/vw/$name/$numName"
if fileExists(srcPath)
if !fileExists(dstPath + "/_SUCCESS")
} {
println(s"$srcPath -> $dstPath")
removeFile(dstPath)
convertFile(srcPath, dstPath)
}
}
println("Use 'doDataConversion' to start data conversion.")
================================================
FILE: scripts/conversion/sampleLibsvm.scala
================================================
type Data = (org.apache.spark.rdd.RDD[String], Long)
def sample(data: Data, n: Long): Data = {
val rdd = data._1
val count = data._2
val ratio = n.toDouble / count
val sampledRdd = rdd.sample(false, scala.math.min(1.0, ratio * 1.01), scala.util.Random.nextLong)
val exactSampledRdd = sampledRdd.zipWithIndex.filter { case (_, i) => i < n } map (_._1)
val exactCount = exactSampledRdd.count
(exactSampledRdd, exactCount)
}
def load(path: String): (Data, Data) = {
val filesAndStats = 0 to 23 map {
case i => {
val dayPath = s"$path/day_${i}.*"
val rdd = sc.textFile(dayPath)
val count = rdd.count
(rdd, count)
}
}
val test = filesAndStats.last
val train = filesAndStats.init.reduce((a, b) => (a._1 union b._1, a._2 + b._2))
(train, test)
}
val fs = org.apache.hadoop.fs.FileSystem.get(sc.hadoopConfiguration)
val numberOfParts = 1024
def writeSamples(data: Data, samples: List[Long], path: String, ext: String): Unit = samples foreach {
case n =>
val name = n.toString.reverse.replaceAll("000", "k").reverse
val writePath = s"$path/$name.$ext"
val hadoopSuccessPath = new org.apache.hadoop.fs.Path(writePath + "/_SUCCESS")
if (fs.exists(hadoopSuccessPath)) {
println(s"Data was already successfully written to $writePath, skipping.")
} else {
val hadoopPath = new org.apache.hadoop.fs.Path(writePath)
println(s"Removing $writePath.")
fs.delete(hadoopPath)
println("Sampling data")
val sampledData = sample(data, n)
println(s"Writing ${sampledData._2} lines to $writePath.")
sampledData._1.coalesce(numberOfParts).saveAsTextFile(writePath)
}
}
def powTenAndTriple(n: Int): List[Long] = { val v = scala.math.pow(10, n).longValue; List(v, 3 * v) }
val testSamples = List(1000000l)
val trainSamples = (4 to 9 flatMap powTenAndTriple).toList
def processDataPersist(what: String): Unit = {
println(s"Working with $what.")
val dataPath = s"criteo_1tb/$what"
println(s"Loading data from $dataPath.")
val (train, test) = load(dataPath)
println("Data loaded.")
def processDataSet(name: String, data: Data, samples: List[Long]): Unit = {
println(s"Sampling $name to ${samples.mkString("[", ", ", "]")} lines.")
writeSamples(data, samples, s"$dataPath/$name", what)
}
test._1.persist
processDataSet("test", test, testSamples)
test._1.unpersist(true)
train._1.persist
processDataSet("train", train, trainSamples)
train._1.unpersist(true)
println(s"Done with $what.")
}
def doDataPreparationLibSVM = {
processDataPersist("libsvm")
}
println("Use 'doDataPreparationLibSVM' to start data preparation.")
================================================
FILE: scripts/running/build_plots.sh
================================================
#!/usr/bin/env bash
python plots.py -i metrics.old.tsv -n 'why_optimize' -c 'b,k,y'
python plots.py -i metrics.selection.tsv -n 'cluster_selection' -c 'k,r,g,b'
python plots.py -i vw_xgb.tsv -n 'local' -p -c 'b,k,y'
python plots.py -i metrics.lr_hash_size.tsv -n 'lr_hash_size' -c 'r,g,b,k,y'
python plots.py -i metrics.cluster.tsv -n 'cluster' -c 'r,g,b,k,y'
python plots.py -i metrics.tsv -n 'local_and_cluster' -c 'r,g,b,k,y' -l '0.13,0.15'
================================================
FILE: scripts/running/measure.py
================================================
import sys
from sklearn.metrics import (
auc,
log_loss,
roc_curve,
)
engine = sys.argv[1]
train_file = sys.argv[2]
test_file = sys.argv[3]
scores_file = test_file + '.predictions'
time_file = train_file + '.time'
def get_last_in_line(s):
return s.rstrip().split( )[-1]
def parse_elapsed_time(s):
return reduce(lambda a, b: a * 60 + b, map(float, get_last_in_line(s).split(':')))
def parse_max_memory(s):
return int(get_last_in_line(s)) * 1024
def parse_cpu(s):
return float(get_last_in_line(s).rstrip('%')) / 100
elapsed = -1
memory = -1
cpu = -1
with open(time_file, 'rb') as f:
for line in f:
if 'Elapsed (wall clock) time' in line:
elapsed = parse_elapsed_time(line)
elif 'Maximum resident set size' in line:
memory = parse_max_memory(line)
elif 'Percent of CPU' in line:
cpu = parse_cpu(line)
with open(test_file, 'rb') as f:
labels = [line.rstrip().split(' ')[0] == '1' for line in f]
with open(scores_file, 'rb') as f:
scores = [float(line.rstrip().split(' ')[0]) for line in f]
fpr, tpr, _ = roc_curve(labels, scores)
roc_auc = auc(fpr, tpr)
ll = log_loss(labels, scores)
try:
train_size = int(train_file.split('/')[-1].split('.')[2].replace('k', '000'))
except:
train_size = 0
print '\t'.join(map(str, [engine, train_size, roc_auc, ll, elapsed, memory, cpu]))
================================================
FILE: scripts/running/plots.py
================================================
from __future__ import print_function
import argparse
import re
import cycler
import pandas
from matplotlib import pyplot
def extract_data_for_plotting(df, what):
return reduce(
lambda left, right: pandas.merge(
left,
right,
how='outer',
on='Train size',
),
map(
lambda name: (
df[df.Engine == name][['Train size', what]]
.rename(columns={what: name})
),
df.Engine.unique(),
),
)
def plot_stuff(df, what, ylabel=None, **kwargs):
data = extract_data_for_plotting(df, what).set_index('Train size')
ax = data.plot(
figsize=(6, 6),
title=what,
grid=True,
linewidth=2.0,
marker='o',
**kwargs
)
ax.legend(loc='best')
if ylabel is not None:
ax.set_ylabel(ylabel)
ax.grid(which='major', linestyle='-')
ax.grid(which='minor', linestyle=':')
what_normalized = re.sub(r'\s', '_', what).lower()
if experiment_name is not None:
ax.get_figure().savefig(what_normalized + '.' + experiment_name + '.png')
else:
ax.get_figure().savefig(what_normalized + '.png')
parser = argparse.ArgumentParser()
parser.add_argument('-i', '--input', type=str, default='metrics.tsv',
help='input file')
parser.add_argument('-n', '--name', type=str,
help='experiment name')
parser.add_argument('-p', '--perf', action='store_true',
help='build perf graphs')
parser.add_argument('-c', '--colors', type=str,
help='color cycle')
parser.add_argument('-l', '--logloss',
help='log loss scale')
args = parser.parse_args()
metrics_file = args.input
experiment_name = args.name
perf_graphs = args.perf
if args.colors is not None:
color_cycle = args.colors.split(',')
pyplot.rc('axes', prop_cycle=(cycler.cycler('color', color_cycle)))
df = (
pandas
.read_csv(metrics_file, sep='\t')
.sort_values(by=['Engine', 'Train size'])
)
plot_stuff(df, 'ROC AUC', logx=True)
if args.logloss is not None:
ll_from, ll_to = map(float, args.logloss.split(','))
plot_stuff(df, 'Log loss', logx=True, ylim=(ll_from, ll_to))
else:
plot_stuff(df, 'Log loss', logx=True)
plot_stuff(df, 'Train time', loglog=True, ylabel='s')
if perf_graphs:
plot_stuff(df, 'Maximum memory', loglog=True, ylabel='bytes')
plot_stuff(df, 'CPU load', logx=True)
================================================
FILE: scripts/running/run.sh
================================================
#!/usr/bin/env bash
DATA_PREFIX="./data"
test_vw="${DATA_PREFIX}/data.test.1kk.vw"
test_xgb="${DATA_PREFIX}/data.test.1kk.libsvm"
for train_num in 10k 30k 100k 300k 1kk 3kk 10kk 30kk 100kk 300kk 1kkk 3kkk; do
echo " * * * Train size ${train_num} lines * * *"
train_vw="${DATA_PREFIX}/data.train.${train_num}.vw"
train_xgb="${DATA_PREFIX}/data.train.${train_num}.libsvm"
echo "Running VW with train ${train_vw} and ${test_vw}"
./vw.sh "${train_vw}" "${test_vw}"
echo "Running XGBoost with train ${train_xgb} and ${test_xgb}"
./xgb.sh "${train_xgb}" "${test_xgb}"
echo "Running XGBoost (out-of-core) with train ${train_xgb} and ${test_xgb}"
./xgb.ooc.sh "${train_xgb}" "${test_xgb}"
done
================================================
FILE: scripts/running/vw.conf
================================================
-b 29
-l 0.3
--initial_t 1
--decay_learning_rate 0.5
--power_t 0.5
--l1 1e-15
--l2 0
================================================
FILE: scripts/running/vw.sh
================================================
#!/usr/bin/env bash
TRAIN="${1}"
TEST="${2}"
if [[ "${TRAIN}" == "" || "${TEST}" == "" ]]; then
echo "Usage: $0 train test"
exit 1
fi
TIME="${TRAIN}.time"
MODEL="${TRAIN}.model"
PREDICTIONS="${TEST}.predictions"
VW_OPTS=($(cat vw.conf))
echo VW_OPTS = "${VW_OPTS[@]}"
/usr/local/bin/time -v --output="${TIME}" \
vw83 --link=logistic --loss_function=logistic -d "${TRAIN}" -f "${MODEL}" "${VW_OPTS[@]}"
vw83 -i "${MODEL}" --loss_function=logistic -t -d "${TEST}" -p "${PREDICTIONS}"
METRICS="metrics.tsv"
if ! [[ -e ${METRICS} ]]; then
echo -e "Engine\tTrain size\tROC AUC\tLog loss\tTrain time\tMaximum memory\tCPU load" | tee "${METRICS}"
fi
python measure.py "vw" "${TRAIN}" "${TEST}" | tee -a "${METRICS}"
rm "${TIME}" "${PREDICTIONS}"
================================================
FILE: scripts/running/xgb.conf
================================================
booster = gbtree
objective = binary:logistic
nthread = 12
eval_metric = logloss
max_depth = 7
num_round = 200
eta = 0.2
gamma = 0.4
subsample = 0.8
colsample_bytree = 0.8
min_child_weight = 20
alpha = 3
lambda = 100
================================================
FILE: scripts/running/xgb.ooc.sh
================================================
#!/usr/bin/env bash
TRAIN="${1}"
TEST="${2}"
TRAIN_OOC="${TRAIN}#${TRAIN}.cache"
if [[ "${TRAIN}" == "" || "${TEST}" == "" ]]; then
echo "Usage: $0 train test"
exit 1
fi
TIME="${TRAIN}.time"
MODEL="${TRAIN}.ooc.model"
PREDICTIONS="${TEST}.predictions"
/usr/local/bin/time -v --output="${TIME}" \
xgboost xgb.conf data="${TRAIN_OOC}" model_out="${MODEL}"
xgboost xgb.conf task=pred test:data="${TEST}" model_in="${MODEL}" name_pred="${PREDICTIONS}"
METRICS="metrics.tsv"
if ! [[ -e ${METRICS} ]]; then
echo -e "Engine\tTrain size\tROC AUC\tLog loss\tTrain time\tMaximum memory\tCPU load" | tee "${METRICS}"
fi
python measure.py "xgb.ooc" "${TRAIN}" "${TEST}" | tee -a "${METRICS}"
rm "${TIME}" "${PREDICTIONS}" "${TRAIN}.cache"* "${TEST}.buffer"
================================================
FILE: scripts/running/xgb.sh
================================================
#!/usr/bin/env bash
TRAIN="${1}"
TEST="${2}"
if [[ "${TRAIN}" == "" || "${TEST}" == "" ]]; then
echo "Usage: $0 train test"
exit 1
fi
TIME="${TRAIN}.time"
MODEL="${TRAIN}.model"
PREDICTIONS="${TEST}.predictions"
/usr/local/bin/time -v --output="${TIME}" \
xgboost xgb.conf data="${TRAIN}" model_out="${MODEL}"
xgboost xgb.conf task=pred test:data="${TEST}" model_in="${MODEL}" name_pred="${PREDICTIONS}"
METRICS="metrics.tsv"
if ! [[ -e ${METRICS} ]]; then
echo -e "Engine\tTrain size\tROC AUC\tLog loss\tTrain time\tMaximum memory\tCPU load" | tee "${METRICS}"
fi
python measure.py "xgb" "${TRAIN}" "${TEST}" | tee -a "${METRICS}"
rm "${TIME}" "${PREDICTIONS}" "${TRAIN}.buffer" "${TEST}.buffer"
gitextract_hh2ek1q9/
├── README.md
├── notebooks/
│ ├── experiment_local.ipynb
│ ├── experiment_spark_lr.ipynb
│ └── experiment_spark_rf.ipynb
├── results/
│ ├── metrics.cluster.tsv
│ ├── metrics.lr_hash_size.tsv
│ ├── metrics.old.tsv
│ ├── metrics.selection.tsv
│ ├── metrics.tsv
│ └── vw_xgb.tsv
└── scripts/
├── conversion/
│ ├── criteoToLibsvm.scala
│ ├── libsvmToVw.scala
│ └── sampleLibsvm.scala
└── running/
├── build_plots.sh
├── measure.py
├── plots.py
├── run.sh
├── vw.conf
├── vw.sh
├── xgb.conf
├── xgb.ooc.sh
└── xgb.sh
SYMBOL INDEX (6 symbols across 2 files) FILE: scripts/running/measure.py function get_last_in_line (line 17) | def get_last_in_line(s): function parse_elapsed_time (line 20) | def parse_elapsed_time(s): function parse_max_memory (line 23) | def parse_max_memory(s): function parse_cpu (line 26) | def parse_cpu(s): FILE: scripts/running/plots.py function extract_data_for_plotting (line 12) | def extract_data_for_plotting(df, what): function plot_stuff (line 30) | def plot_stuff(df, what, ylabel=None, **kwargs):
Condensed preview — 22 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (112K chars).
[
{
"path": "README.md",
"chars": 13991,
"preview": "# Criteo 1 TiB benchmark\n\n\n\n## Table of contents\n\n* [Introduction](#introduction)\n* [Task and data](#task-and-data)\n* [A"
},
{
"path": "notebooks/experiment_local.ipynb",
"chars": 30728,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Criteo 1 TiB benchmark\\n\",\n \"\\"
},
{
"path": "notebooks/experiment_spark_lr.ipynb",
"chars": 17171,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"collapsed\": true\n },\n \"source\": [\n \"# Criteo "
},
{
"path": "notebooks/experiment_spark_rf.ipynb",
"chars": 16201,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Criteo 1 TiB benchmark - Spark.ML"
},
{
"path": "results/metrics.cluster.tsv",
"chars": 1307,
"preview": "\"Engine\"\t\"Train size\"\t\"ROC AUC\"\t\"Log loss\"\t\"Train time\"\n\"lr\"\t10000\t0.577664202812\t0.191408441626\t50.9623498917\n\"lr\"\t3000"
},
{
"path": "results/metrics.lr_hash_size.tsv",
"chars": 1299,
"preview": "\"Engine\"\t\"Train size\"\t\"ROC AUC\"\t\"Log loss\"\t\"Train time\"\n\"lr, 100k hashes\"\t10000\t0.577664202812\t0.191408441626\t50.9623498"
},
{
"path": "results/metrics.old.tsv",
"chars": 687,
"preview": "\"Engine\"\t\"Train size\"\t\"ROC AUC\"\t\"Log loss\"\t\"Train time\"\n\"vw\"\t10000\t0.609\t0.1501\t1\n\"vw\"\t30000\t0.632\t0.1492\t2\n\"vw\"\t100000\t"
},
{
"path": "results/metrics.selection.tsv",
"chars": 1380,
"preview": "\"Engine\"\t\"Train size\"\t\"ROC AUC\"\t\"Log loss\"\t\"Train time\"\nlr\t10000\t0.578177884844\t0.178971884565\t29.5085098743\nrf\t10000\t0."
},
{
"path": "results/metrics.tsv",
"chars": 2836,
"preview": "\"Engine\"\t\"Train size\"\t\"ROC AUC\"\t\"Log loss\"\t\"Train time\"\n\"vw\"\t10000\t0.617455161515\t0.144524706901\t15.17\n\"vw\"\t30000\t0.6472"
},
{
"path": "results/vw_xgb.tsv",
"chars": 2083,
"preview": "\"Engine\"\t\"Train size\"\t\"ROC AUC\"\t\"Log loss\"\t\"Train time\"\t\"Maximum memory\"\t\"CPU load\"\n\"vw\"\t10000\t0.617455161515\t0.14452470"
},
{
"path": "scripts/conversion/criteoToLibsvm.scala",
"chars": 1083,
"preview": "def rowToLibsvm(row: org.apache.spark.sql.Row): String = {\n 0 until row.length flatMap {\n case 0 => Some(row(0).toSt"
},
{
"path": "scripts/conversion/libsvmToVw.scala",
"chars": 1653,
"preview": "import java.security.MessageDigest\n\nimport org.apache.hadoop.fs.{FileSystem, Path => HadoopPath}\nimport org.apache.spark"
},
{
"path": "scripts/conversion/sampleLibsvm.scala",
"chars": 2669,
"preview": "type Data = (org.apache.spark.rdd.RDD[String], Long)\n\ndef sample(data: Data, n: Long): Data = {\n val rdd = data._1\n va"
},
{
"path": "scripts/running/build_plots.sh",
"chars": 445,
"preview": "#!/usr/bin/env bash\n\npython plots.py -i metrics.old.tsv -n 'why_optimize' -c 'b,k,y'\npython plots.py -i metrics.selectio"
},
{
"path": "scripts/running/measure.py",
"chars": 1394,
"preview": "import sys\nfrom sklearn.metrics import (\n auc,\n log_loss,\n roc_curve,\n)\n\n\nengine = sys.argv[1]\ntrain_file = sys"
},
{
"path": "scripts/running/plots.py",
"chars": 2509,
"preview": "from __future__ import print_function\n\nimport argparse\nimport re\n\nimport cycler\nimport pandas\n\nfrom matplotlib import py"
},
{
"path": "scripts/running/run.sh",
"chars": 730,
"preview": "#!/usr/bin/env bash\n\n\nDATA_PREFIX=\"./data\"\n\ntest_vw=\"${DATA_PREFIX}/data.test.1kk.vw\"\ntest_xgb=\"${DATA_PREFIX}/data.test"
},
{
"path": "scripts/running/vw.conf",
"chars": 121,
"preview": "-b 29\n-l 0.3\n--initial_t 1\n--decay_learning_rate 0.5\n--power_t 0.5\n--l1 1e-15\n--l2 0"
},
{
"path": "scripts/running/vw.sh",
"chars": 767,
"preview": "#!/usr/bin/env bash\n\n\nTRAIN=\"${1}\"\nTEST=\"${2}\"\n\nif [[ \"${TRAIN}\" == \"\" || \"${TEST}\" == \"\" ]]; then\n echo \"Usage: $0 t"
},
{
"path": "scripts/running/xgb.conf",
"chars": 228,
"preview": "booster = gbtree\nobjective = binary:logistic\nnthread = 12\neval_metric = logloss\n\nmax_depth = 7\nnum_round = 200\neta = "
},
{
"path": "scripts/running/xgb.ooc.sh",
"chars": 773,
"preview": "#!/usr/bin/env bash\n\n\nTRAIN=\"${1}\"\nTEST=\"${2}\"\n\nTRAIN_OOC=\"${TRAIN}#${TRAIN}.cache\"\n\nif [[ \"${TRAIN}\" == \"\" || \"${TEST}\""
},
{
"path": "scripts/running/xgb.sh",
"chars": 724,
"preview": "#!/usr/bin/env bash\n\n\nTRAIN=\"${1}\"\nTEST=\"${2}\"\n\nif [[ \"${TRAIN}\" == \"\" || \"${TEST}\" == \"\" ]]; then\n echo \"Usage: $0 t"
}
]
About this extraction
This page contains the full source code of the rambler-digital-solutions/criteo-1tb-benchmark GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 22 files (98.4 KB), approximately 31.6k tokens, and a symbol index with 6 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.