![\\\"Model](\\\"https://github.com/JoeriHermans/dist-keras/blob/master/resources/model_parallelism.png?raw=true\\\")
\\n\",\n \"\\n\",\n \"### Data parallelism\\n\",\n \"\\n\",\n \"As stated in the introduction, in order to train a large network in a reasonable amount of time, we need to parallize the optimization process (which is the learning of the model). In this settings, we take *several* model replicas, and distribute them over multiple machines. Of course, it would also be possible to combine this with the model parallelism approach. However, for the sake of simplicity, let us assume that a model (or several models) can be contained on a single machine. In order to parallelize the training, and to improve the usage of the resources of the cluster, we distribute the models over several machines.\\n\",\n \"\\n\",\n \"In order to build a distributed learning scheme using data parallelism, you would in the most simple case need at least 1 **parameter server**. A parameter server is basically a thread (or a collection of threads) which aggregate the incoming gradient updates of the workers into a so-called center variable, which acts as a *global consensus* variable. Finally, the weights which are stored in the center variable will eventually be used by the produced model.\\n\",\n \"\\n\",\n \"![\\\"Data](\\\"https://github.com/JoeriHermans/dist-keras/blob/master/resources/data_parallelism.png?raw=true\\\")
\\n\",\n \"\\n\",\n \"There are two general approaches towards solving data parallelism. The most straightforward is a **synchronous** method. In short, a synchronous data parallel method will wait for all workers to finish the current mini-batch or stochastic sample before continuing to the next iteration. Synchronous methods have the advantage that all workers will use the most recent center variable, i.e., a worker knows that all other workers will use the same center variable. However, the main disadvantage of this method is the synchronization itself. A synchronous method will never be truly synchronous. This is due to the many, and possibly different, machines. Furthermore, every machine could have a different workload, which would influence the training speed of a worker. As a result, synchronous methods need additional waiting mechanisms to synchronize all workers. These locking mechanisms will make sure that that all workers compute the next gradient based on the same center variable. However, locking mechanisms induce a significant wait which will significantly influence the training speed. For example, imagine a cluster node with an unusual high load. This high load will, due to CPU sharing, cause the training procedure to slow down. Which in turn, will cause the other workers to wait for this single node. Of course, this is just a simple and possibly extreme example, but this example shows how a single worker could significantly influence the training time of all workers.\\n\",\n \"\\n\",\n \"![\\\"Data](\\\"https://github.com/JoeriHermans/dist-keras/blob/master/resources/synchronous_method.png?raw=true\\\")
\\n\",\n \"\\n\",\n \"A very simple, but radical \\\"solution\\\" for this synchronization problem is to not synchronize the workers :) Workers simply fetch the center variable and update the parameter server with the computed gradient whenever a worker is ready. This approach is called an **asynchronous** data parallel method. Asynchronous methods, compared to synchronous methods, will have a different set of problems. One of these is a so-called **stale gradient**. This a gradient based on an older version of the center variable while the current center variable is a center variable which has already been updated by other workers. One approach to solve this is to induce and exponential decay factor to the gradient updates. However, this would waste computational resources, but of course, one could just get the most recent weights from the parameter server and then start again. However, as we will show later, it is actually stale gradients (result of asynchrony) that induce *implicit momentum* to the learning process [[2]](https://arxiv.org/pdf/1606.04487v4.pdf).\\n\",\n \"\\n\",\n \"![\\\"Data](\\\"https://github.com/JoeriHermans/dist-keras/blob/master/resources/asynchronous_method.png?raw=true\\\")
\\n\",\n \"\\n\",\n \"At this point you probably ask the question: **why does this actually work?** A lot of people suggest this is due to the sparsity of the gradients. Intuitively, image having multiple workers processing different data (since every worker has its own dat partition), chances are the weights updates will be totally dissimilar since we are training a large network with a lot of tunable parameters. Furthermore, techniques such as dropout (if they are applied differently among the replicas) only increase the sparsity updates3.\\n\",\n \"\\n\",\n \"### Formalization\\n\",\n \"\\n\",\n \"We would also like to inform the reader that the general problem to be solved is the so-called **global consensus optimization** problem. A popular approach towards solving this is using the Alternating Direction Method of Multipliers (ADMM) [[3]](http://www.jmlr.org/proceedings/papers/v32/zhange14.pdf) [[4]](http://web.stanford.edu/~boyd/papers/pdf/admm_distr_stats.pdf). However, since this is outside the scope of this notebook we will not review this in-depth. But, we would like to note that the *Elastic Averaging* methods [[5]](https://arxiv.org/pdf/1412.6651.pdf) by Zhang et al., which we included in *Distributed Keras* are based on ADMM.\\n\",\n \"\\n\",\n \"**1:** Hybrids are possible as well. \\n\",\n \"**2:** This is mainly used for the computation of the network outputs [[1]](https://papers.nips.cc/paper/4687-large-scale-distributed-deep-networks.pdf). \\n\",\n \"**3:** A way to check the sparsity between 2 gradients is to put all the weights in to a 1 dimensional vector, and then compute the [cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity).\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Distributed Keras\\n\",\n \"\\n\",\n \"Distributed Keras is a framework which uses Apache Spark and Keras. We chose for Spark because of the distributed environment. This allows us to preprocess the data in a distributed manner, and train our deep learning models on the same architecture, while still having the modeling simplicity of Keras.\\n\",\n \"\\n\",\n \"### Architecture\\n\",\n \"\\n\",\n \"Our architecture is very similar to the architecture discussed in [[1]](https://papers.nips.cc/paper/4687-large-scale-distributed-deep-networks.pdf). However, we employ Apache Spark for data parallel reading and handling larger than memory datasets. The parameter server will always be created in the **Spark Driver**. This **is the program which creates the Spark Context**. For example, if the Jupyter installation of this notebook is running on the Spark cluster, then a cluster node will host the parameter server. However, if you run a Python script, which connects to a remote Spark cluster, then your computer will run the Spark Driver, and as a result will run the parameter server. In that case, be sure your network connection is able to handle the load, else your computer will be the bottleneck in the learning process.\\n\",\n \"\\n\",\n \"![\\\"Model](\\\"https://github.com/JoeriHermans/dist-keras/blob/master/resources/distkeras_architecture.png?raw=true\\\")
\\n\",\n \"\\n\",\n \"### Implementation of costum distributed optimizer\\n\",\n \"\\n\",\n \"In order to implement your own optimizer you need 2 classes. First, define your optimizer using the *Trainer* interface. We already supplied an *AsynchronousDistributedTrainer*, and an *SynchronousDistributedTrainer*. However, if you require an other procedure, please feel free to do so. Finally, you need a worker class. This class must have a *train* method with the required arguments, as specified by Apache Spark.\\n\",\n \"\\n\",\n \"### Usage\\n\",\n \"\\n\",\n \"In the following sections, we will give you an example how a complete workflow will look like. This includes setting up a Spark context, reading, preprocessing, and normalizing the data. Finally, we create a relatively simple model (feel free to adjust the parameters) with Keras and optimize it using the different distributed optimizers which are included by default.\\n\",\n \"\\n\",\n \"#### Dataset\\n\",\n \"\\n\",\n \"We are using the ATLAS Higgs dataset constructed for the Kaggle machine learning challenge. This dataset is quite limited, it contains only **250000** instances. 40% of which we will be using as a test set. For future experiments, it would be usefull to integrate well understood datasets such as CIFAR or MNIST to evaluate against other optimizers. However, it would be nice to have a \\\"well understood\\\" HEP (High Energy Physics) dataset for this task :) \"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 2,\n \"metadata\": {\n \"collapsed\": false\n },\n \"outputs\": [\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"Using TensorFlow backend.\\n\"\n ]\n }\n ],\n \"source\": [\n \"import numpy as np\\n\",\n \"\\n\",\n \"import time\\n\",\n \"\\n\",\n \"import requests\\n\",\n \"\\n\",\n \"from keras.optimizers import *\\n\",\n \"from keras.models import Sequential\\n\",\n \"from keras.layers.core import Dense, Dropout, Activation\\n\",\n \"\\n\",\n \"from pyspark import SparkContext\\n\",\n \"from pyspark import SparkConf\\n\",\n \"\\n\",\n \"from pyspark.ml.feature import StandardScaler\\n\",\n \"from pyspark.ml.feature import VectorAssembler\\n\",\n \"from pyspark.ml.feature import StringIndexer\\n\",\n \"from pyspark.ml.evaluation import MulticlassClassificationEvaluator\\n\",\n \"from pyspark.mllib.evaluation import BinaryClassificationMetrics\\n\",\n \"\\n\",\n \"from distkeras.trainers import *\\n\",\n \"from distkeras.predictors import *\\n\",\n \"from distkeras.transformers import *\\n\",\n \"from distkeras.evaluators import *\\n\",\n \"from distkeras.utils import *\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 3,\n \"metadata\": {\n \"collapsed\": true\n },\n \"outputs\": [],\n \"source\": [\n \"# Modify these variables according to your needs.\\n\",\n \"application_name = \\\"Distributed Keras Notebook\\\"\\n\",\n \"using_spark_2 = False\\n\",\n \"local = False\\n\",\n \"if local:\\n\",\n \" # Tell master to use local resources.\\n\",\n \" master = \\\"local[*]\\\"\\n\",\n \" num_cores = 3\\n\",\n \" num_executors = 1\\n\",\n \"else:\\n\",\n \" # Tell master to use YARN.\\n\",\n \" master = \\\"yarn-client\\\"\\n\",\n \" num_executors = 6\\n\",\n \" num_cores = 2\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 4,\n \"metadata\": {\n \"collapsed\": false\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"Number of desired executors: 6\\n\",\n \"Number of desired cores / executor: 2\\n\",\n \"Total number of workers: 12\\n\"\n ]\n }\n ],\n \"source\": [\n \"# This variable is derived from the number of cores and executors, and will be used to assign the number of model trainers.\\n\",\n \"num_workers = num_executors * num_cores\\n\",\n \"\\n\",\n \"print(\\\"Number of desired executors: \\\" + `num_executors`)\\n\",\n \"print(\\\"Number of desired cores / executor: \\\" + `num_cores`)\\n\",\n \"print(\\\"Total number of workers: \\\" + `num_workers`)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 5,\n \"metadata\": {\n \"collapsed\": true\n },\n \"outputs\": [],\n \"source\": [\n \"import os\\n\",\n \"\\n\",\n \"# Use the DataBricks CSV reader, this has some nice functionality regarding invalid values.\\n\",\n \"os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages com.databricks:spark-csv_2.10:1.4.0 pyspark-shell'\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"#### Preparing a Spark Context\\n\",\n \"\\n\",\n \"In order to read our (big) dataset into our Spark Cluster, we first need a Spark Context. However, since Spark 2.0 there are some changes regarding the initialization of a Spark Context. For example, SQLContext and HiveContext do not have to be initialized separately anymore.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 6,\n \"metadata\": {\n \"collapsed\": true\n },\n \"outputs\": [],\n \"source\": [\n \"conf = SparkConf()\\n\",\n \"conf.set(\\\"spark.app.name\\\", application_name)\\n\",\n \"conf.set(\\\"spark.master\\\", master)\\n\",\n \"conf.set(\\\"spark.executor.cores\\\", `num_cores`)\\n\",\n \"conf.set(\\\"spark.executor.instances\\\", `num_executors`)\\n\",\n \"conf.set(\\\"spark.locality.wait\\\", \\\"0\\\")\\n\",\n \"conf.set(\\\"spark.serializer\\\", \\\"org.apache.spark.serializer.KryoSerializer\\\");\\n\",\n \"\\n\",\n \"# Check if the user is running Spark 2.0 +\\n\",\n \"if using_spark_2:\\n\",\n \" sc = SparkSession.builder.config(conf=conf) \\\\\\n\",\n \" .appName(application_name) \\\\\\n\",\n \" .getOrCreate()\\n\",\n \"else:\\n\",\n \" # Create the Spark context.\\n\",\n \" sc = SparkContext(conf=conf)\\n\",\n \" # Add the missing imports\\n\",\n \" from pyspark import SQLContext\\n\",\n \" sqlContext = SQLContext(sc)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 8,\n \"metadata\": {\n \"collapsed\": false\n },\n \"outputs\": [],\n \"source\": [\n \"# Check if we are using Spark 2.0\\n\",\n \"if using_spark_2:\\n\",\n \" reader = sc\\n\",\n \"else:\\n\",\n \" reader = sqlContext\\n\",\n \"# Read the dataset.\\n\",\n \"raw_dataset = reader.read.format('com.databricks.spark.csv') \\\\\\n\",\n \" .options(header='true', inferSchema='true').load(\\\"data/atlas_higgs.csv\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 9,\n \"metadata\": {\n \"collapsed\": false\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"root\\n\",\n \" |-- EventId: integer (nullable = true)\\n\",\n \" |-- DER_mass_MMC: double (nullable = true)\\n\",\n \" |-- DER_mass_transverse_met_lep: double (nullable = true)\\n\",\n \" |-- DER_mass_vis: double (nullable = true)\\n\",\n \" |-- DER_pt_h: double (nullable = true)\\n\",\n \" |-- DER_deltaeta_jet_jet: double (nullable = true)\\n\",\n \" |-- DER_mass_jet_jet: double (nullable = true)\\n\",\n \" |-- DER_prodeta_jet_jet: double (nullable = true)\\n\",\n \" |-- DER_deltar_tau_lep: double (nullable = true)\\n\",\n \" |-- DER_pt_tot: double (nullable = true)\\n\",\n \" |-- DER_sum_pt: double (nullable = true)\\n\",\n \" |-- DER_pt_ratio_lep_tau: double (nullable = true)\\n\",\n \" |-- DER_met_phi_centrality: double (nullable = true)\\n\",\n \" |-- DER_lep_eta_centrality: double (nullable = true)\\n\",\n \" |-- PRI_tau_pt: double (nullable = true)\\n\",\n \" |-- PRI_tau_eta: double (nullable = true)\\n\",\n \" |-- PRI_tau_phi: double (nullable = true)\\n\",\n \" |-- PRI_lep_pt: double (nullable = true)\\n\",\n \" |-- PRI_lep_eta: double (nullable = true)\\n\",\n \" |-- PRI_lep_phi: double (nullable = true)\\n\",\n \" |-- PRI_met: double (nullable = true)\\n\",\n \" |-- PRI_met_phi: double (nullable = true)\\n\",\n \" |-- PRI_met_sumet: double (nullable = true)\\n\",\n \" |-- PRI_jet_num: integer (nullable = true)\\n\",\n \" |-- PRI_jet_leading_pt: double (nullable = true)\\n\",\n \" |-- PRI_jet_leading_eta: double (nullable = true)\\n\",\n \" |-- PRI_jet_leading_phi: double (nullable = true)\\n\",\n \" |-- PRI_jet_subleading_pt: double (nullable = true)\\n\",\n \" |-- PRI_jet_subleading_eta: double (nullable = true)\\n\",\n \" |-- PRI_jet_subleading_phi: double (nullable = true)\\n\",\n \" |-- PRI_jet_all_pt: double (nullable = true)\\n\",\n \" |-- Weight: double (nullable = true)\\n\",\n \" |-- Label: string (nullable = true)\\n\",\n \"\\n\"\n ]\n }\n ],\n \"source\": [\n \"# Double-check the inferred schema, and get fetch a row to show how the dataset looks like.\\n\",\n \"raw_dataset.printSchema()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"#### Dataset preprocessing and normalization\\n\",\n \"\\n\",\n \"Since Spark's MLlib has some nice features for distributed preprocessing, we made sure we comply to the DataFrame API in order to ensure compatibility. What it basically boils down to, is that all the features (which can have different type) will be aggregated into a single column. More information on Spark MLlib (and other APIs) can be found here: [http://spark.apache.org/docs/latest/ml-guide.html](http://spark.apache.org/docs/latest/ml-guide.html)\\n\",\n \"\\n\",\n \"In the following steps we will show you how to extract the desired columns from the dataset and prepare the for further processing.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 10,\n \"metadata\": {\n \"collapsed\": false\n },\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"[Row(features=DenseVector([138.47, 51.655, 97.827, 27.98, 0.91, 124.711, 2.666, 3.064, 41.928, 197.76, 1.582, 1.396, 0.2, 32.638, 1.017, 0.381, 51.626, 2.273, -2.414, 16.824, -0.277, 258.733, 2.0, 67.435, 2.15, 0.444, 46.062, 1.24, -2.475, 113.497]))]\"\n ]\n },\n \"execution_count\": 10,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"# First, we would like to extract the desired features from the raw dataset.\\n\",\n \"# We do this by constructing a list with all desired columns.\\n\",\n \"features = raw_dataset.columns\\n\",\n \"features.remove('EventId')\\n\",\n \"features.remove('Weight')\\n\",\n \"features.remove('Label')\\n\",\n \"# Next, we use Spark's VectorAssembler to \\\"assemble\\\" (create) a vector of all desired features.\\n\",\n \"# http://spark.apache.org/docs/latest/ml-features.html#vectorassembler\\n\",\n \"vector_assembler = VectorAssembler(inputCols=features, outputCol=\\\"features\\\")\\n\",\n \"# This transformer will take all columns specified in features, and create an additional column \\\"features\\\" which will contain all the desired features aggregated into a single vector.\\n\",\n \"dataset = vector_assembler.transform(raw_dataset)\\n\",\n \"\\n\",\n \"# Show what happened after applying the vector assembler.\\n\",\n \"# Note: \\\"features\\\" column got appended to the end.\\n\",\n \"dataset.select(\\\"features\\\").take(1)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 11,\n \"metadata\": {\n \"collapsed\": true\n },\n \"outputs\": [],\n \"source\": [\n \"# Apply feature normalization with standard scaling. This will transform a feature to have mean 0, and std 1.\\n\",\n \"# http://spark.apache.org/docs/latest/ml-features.html#standardscaler\\n\",\n \"standard_scaler = StandardScaler(inputCol=\\\"features\\\", outputCol=\\\"features_normalized\\\", withStd=True, withMean=True)\\n\",\n \"standard_scaler_model = standard_scaler.fit(dataset)\\n\",\n \"dataset = standard_scaler_model.transform(dataset)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 12,\n \"metadata\": {\n \"collapsed\": false\n },\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"[Row(Label=u's', label_index=1.0),\\n\",\n \" Row(Label=u'b', label_index=0.0),\\n\",\n \" Row(Label=u'b', label_index=0.0),\\n\",\n \" Row(Label=u'b', label_index=0.0),\\n\",\n \" Row(Label=u'b', label_index=0.0)]\"\n ]\n },\n \"execution_count\": 12,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"# If we look at the dataset, the Label column consists of 2 entries, i.e., b (background), and s (signal).\\n\",\n \"# Our neural network will not be able to handle these characters, so instead, we convert it to an index so we can indicate that output neuron with index 0 is background, and 1 is signal.\\n\",\n \"# http://spark.apache.org/docs/latest/ml-features.html#stringindexer\\n\",\n \"label_indexer = StringIndexer(inputCol=\\\"Label\\\", outputCol=\\\"label_index\\\").fit(dataset)\\n\",\n \"dataset = label_indexer.transform(dataset)\\n\",\n \"\\n\",\n \"# Show the result of the label transformation.\\n\",\n \"dataset.select(\\\"Label\\\", \\\"label_index\\\").take(5)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 13,\n \"metadata\": {\n \"collapsed\": true\n },\n \"outputs\": [],\n \"source\": [\n \"# Define some properties of the neural network for later use.\\n\",\n \"nb_classes = 2 # Number of output classes (signal and background)\\n\",\n \"nb_features = len(features)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"collapsed\": false,\n \"scrolled\": true\n },\n \"outputs\": [],\n \"source\": [\n \"# We observe that Keras is not able to work with these indexes.\\n\",\n \"# What it actually expects is a vector with an identical size to the output layer.\\n\",\n \"# Our framework provides functionality to do this with ease.\\n\",\n \"# What it basically does, given an expected vector dimension, \\n\",\n \"# it prepares zero vector with the specified dimensionality, and will set the neuron\\n\",\n \"# with a specific label index to one. (One-Hot encoding)\\n\",\n \"\\n\",\n \"# For example:\\n\",\n \"# 1. Assume we have a label index: 3\\n\",\n \"# 2. Output dimensionality: 5\\n\",\n \"# With these parameters, we obtain the following vector in the DataFrame column: [0,0,0,1,0]\\n\",\n \"\\n\",\n \"transformer = OneHotTransformer(output_dim=nb_classes, input_col=\\\"label_index\\\", output_col=\\\"label\\\")\\n\",\n \"dataset = transformer.transform(dataset)\\n\",\n \"# Only select the columns we need (less data shuffling) while training.\\n\",\n \"dataset = dataset.select(\\\"features_normalized\\\", \\\"label_index\\\", \\\"label\\\")\\n\",\n \"\\n\",\n \"# Show the expected output vectors of the neural network.\\n\",\n \"dataset.select(\\\"label_index\\\", \\\"label\\\").take(1)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"**Warning**: shuffling on a large dataset will take some time.\\n\",\n \"\\n\",\n \"We recommend users to first preprocess and shuffle their data, as is described in the data preprocessing notebook.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"collapsed\": true\n },\n \"outputs\": [],\n \"source\": [\n \"# Shuffle the dataset.\\n\",\n \"dataset = shuffle(dataset)\\n\",\n \"\\n\",\n \"# Note: we also support shuffling in the trainers by default.\\n\",\n \"# However, since this would require a shuffle for every training we will only do it once here.\\n\",\n \"# If you want, you can enable the training shuffling by specifying shuffle=True in the train() function.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"collapsed\": false\n },\n \"outputs\": [],\n \"source\": [\n \"# Finally, we create a trainingset and a testset.\\n\",\n \"(training_set, test_set) = dataset.randomSplit([0.6, 0.4])\\n\",\n \"training_set.cache()\\n\",\n \"test_set.cache()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"#### Model construction\\n\",\n \"\\n\",\n \"We will now construct a relatively simple Keras model (without any modifications) which, hopefully, will be able to classify the dataset.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"collapsed\": false\n },\n \"outputs\": [],\n \"source\": [\n \"model = Sequential()\\n\",\n \"model.add(Dense(500, input_shape=(nb_features,)))\\n\",\n \"model.add(Activation('relu'))\\n\",\n \"model.add(Dropout(0.4))\\n\",\n \"model.add(Dense(500))\\n\",\n \"model.add(Activation('relu'))\\n\",\n \"model.add(Dense(nb_classes))\\n\",\n \"model.add(Activation('softmax'))\\n\",\n \"\\n\",\n \"model.summary()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"##### Worker Optimizer and Loss\\n\",\n \"\\n\",\n \"In order to evaluate the gradient on the model replicas, we have to specify an optimizer and a loss method. For this, we just follow the Keras API as defined in the documentation: [https://keras.io/optimizers/](https://keras.io/optimizers/) and [https://keras.io/objectives/](https://keras.io/objectives/).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"collapsed\": true\n },\n \"outputs\": [],\n \"source\": [\n \"optimizer = 'adagrad'\\n\",\n \"loss = 'categorical_crossentropy'\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"#### Training\\n\",\n \"\\n\",\n \"In the following cells we will train and evaluate the model using different distributed trainers, however, we will as well provide a baseline metric using a **SingleTrainer**, which is basically an instance of the Adagrad optimizer running on Spark.\\n\",\n \"\\n\",\n \"Furthermore, we will also evaluate every training using Spark's MulticlassClassificationEvaluator [https://spark.apache.org/docs/latest/api/python/pyspark.ml.html#pyspark.ml.evaluation.MulticlassClassificationEvaluator](https://spark.apache.org/docs/latest/api/python/pyspark.ml.html#pyspark.ml.evaluation.MulticlassClassificationEvaluator).\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"##### Evaluation\\n\",\n \"\\n\",\n \"We will evaluate all algorithms using the F1 [https://en.wikipedia.org/wiki/F1_score](https://en.wikipedia.org/wiki/F1_score) and accuracy metric.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"collapsed\": true\n },\n \"outputs\": [],\n \"source\": [\n \"def evaluate_accuracy(model):\\n\",\n \" global test_set\\n\",\n \" \\n\",\n \" # Allocate a Distributed Keras Accuracy evaluator.\\n\",\n \" evaluator = AccuracyEvaluator(prediction_col=\\\"prediction_index\\\", label_col=\\\"label_index\\\")\\n\",\n \" # Clear the prediction column from the testset.\\n\",\n \" test_set = test_set.select(\\\"features_normalized\\\", \\\"label_index\\\", \\\"label\\\")\\n\",\n \" # Apply a prediction from a trained model.\\n\",\n \" predictor = ModelPredictor(keras_model=trained_model, features_col=\\\"features_normalized\\\")\\n\",\n \" test_set = predictor.predict(test_set)\\n\",\n \" # Allocate an index transformer.\\n\",\n \" index_transformer = LabelIndexTransformer(output_dim=nb_classes)\\n\",\n \" # Transform the prediction vector to an indexed label.\\n\",\n \" test_set = index_transformer.transform(test_set)\\n\",\n \" # Fetch the score.\\n\",\n \" score = evaluator.evaluate(test_set)\\n\",\n \" \\n\",\n \" return score\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"collapsed\": true\n },\n \"outputs\": [],\n \"source\": [\n \"def add_result(trainer, accuracy, dt):\\n\",\n \" global results;\\n\",\n \" \\n\",\n \" # Store the metrics.\\n\",\n \" results[trainer] = {}\\n\",\n \" results[trainer]['accuracy'] = accuracy;\\n\",\n \" results[trainer]['time_spent'] = dt\\n\",\n \" # Display the metrics.\\n\",\n \" print(\\\"Trainer: \\\" + str(trainer))\\n\",\n \" print(\\\" - Accuracy: \\\" + str(accuracy))\\n\",\n \" print(\\\" - Training time: \\\" + str(dt))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"But first, we will allocate a simple datastructure which will hold the results.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"collapsed\": true\n },\n \"outputs\": [],\n \"source\": [\n \"results = {}\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"##### SingleTrainer\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"A ***SingleTrainer*** is used as a benchmarking trainer to compare to distributed trainer. However, one could also use this trainer if the dataset is too big to fit in memory.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"collapsed\": false\n },\n \"outputs\": [],\n \"source\": [\n \"trainer = SingleTrainer(keras_model=model, worker_optimizer=optimizer,\\n\",\n \" loss=loss, features_col=\\\"features_normalized\\\",\\n\",\n \" label_col=\\\"label\\\", num_epoch=1, batch_size=32)\\n\",\n \"trained_model = trainer.train(training_set)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"collapsed\": false\n },\n \"outputs\": [],\n \"source\": [\n \"# Fetch the evaluation metrics.\\n\",\n \"accuracy = evaluate_accuracy(trained_model)\\n\",\n \"dt = trainer.get_training_time()\\n\",\n \"# Add the metrics to the results.\\n\",\n \"add_result('single', accuracy, dt)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"##### Asynchronous EASGD\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"EASGD based methods, proposed by Zhang et al., transmit the complete parametrization instead of the gradient. These methods will then \\\"average\\\" the difference of the center variable and the backpropagated worker variable. This is used to compute a new master variable, on which the worker nodes will base their backpropagation in the next iteration on.\\n\",\n \"\\n\",\n \"Asynchronous EASGD will do this in an asynchronous fashion, meaning, whenever a worker node is done processing its mini-batch after a certain amount of iterations (communication window), then the computed parameter will be communicated with the parameter server, which will update the center (master) variable immediately without waiting for other workers.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"collapsed\": false\n },\n \"outputs\": [],\n \"source\": [\n \"trainer = AEASGD(keras_model=model, worker_optimizer=optimizer, loss=loss, num_workers=num_workers, \\n\",\n \" batch_size=32, features_col=\\\"features_normalized\\\", label_col=\\\"label\\\", num_epoch=1,\\n\",\n \" communication_window=32, rho=5.0, learning_rate=0.1)\\n\",\n \"trainer.set_parallelism_factor(1)\\n\",\n \"trained_model = trainer.train(training_set)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"collapsed\": false\n },\n \"outputs\": [],\n \"source\": [\n \"# Fetch the evaluation metrics.\\n\",\n \"accuracy = evaluate_accuracy(trained_model)\\n\",\n \"dt = trainer.get_training_time()\\n\",\n \"# Add the metrics to the results.\\n\",\n \"add_result('aeasgd', accuracy, dt)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"##### Asynchronous EAMSGD\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"The only difference between asynchronous EAMSGD and asynchronous EASGD is the possibility of specifying an explicit momentum term.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"collapsed\": true\n },\n \"outputs\": [],\n \"source\": [\n \"trainer = EAMSGD(keras_model=model, worker_optimizer=optimizer, loss=loss, num_workers=num_workers,\\n\",\n \" batch_size=32, features_col=\\\"features_normalized\\\", label_col=\\\"label\\\", num_epoch=1,\\n\",\n \" communication_window=32, rho=5.0, learning_rate=0.1, momentum=0.6)\\n\",\n \"trainer.set_parallelism_factor(1)\\n\",\n \"trained_model = trainer.train(training_set)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"collapsed\": false\n },\n \"outputs\": [],\n \"source\": [\n \"# Fetch the evaluation metrics.\\n\",\n \"accuracy = evaluate_accuracy(trained_model)\\n\",\n \"dt = trainer.get_training_time()\\n\",\n \"# Add the metrics to the results.\\n\",\n \"add_result('eamsgd', accuracy, dt)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"##### DOWNPOUR SGD\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"collapsed\": false\n },\n \"outputs\": [],\n \"source\": [\n \"trainer = DOWNPOUR(keras_model=model, worker_optimizer=optimizer, loss=loss, num_workers=num_workers,\\n\",\n \" batch_size=32, communication_window=5, learning_rate=0.05, num_epoch=1,\\n\",\n \" features_col=\\\"features_normalized\\\", label_col=\\\"label\\\")\\n\",\n \"trainer.set_parallelism_factor(1)\\n\",\n \"trained_model = trainer.train(training_set)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"collapsed\": false\n },\n \"outputs\": [],\n \"source\": [\n \"# Fetch the evaluation metrics.\\n\",\n \"accuracy = evaluate_accuracy(trained_model)\\n\",\n \"dt = trainer.get_training_time()\\n\",\n \"# Add the metrics to the results.\\n\",\n \"add_result('downpour', accuracy, dt)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Experimental observations\\n\",\n \"\\n\",\n \"- DOWNPOUR converges well when a small communication window is used $< 5$.\\n\",\n \"- EASGD based methods on the other hand, thrive using large communication windows $> 25$.\\n\",\n \"- Asynchronous methods induce implicit momentum.\\n\",\n \"\\n\",\n \"## Summary\\n\",\n \"\\n\",\n \"Distributed Deep Learning can **significantly speedup** the learning process. We provide such a framework built on top of Keras and Apache Spark. The latter provides a nice framework for distributed data processing and model evaluation. We can easily integrate our workflows using Apache Spark, and thus speeding up the **data preprocessing**, and our **model optimization procedure** while still having the same **modelling simplicity**.\\n\",\n \"\\n\",\n \"Our group is always open on further collaboration on this work, and would like to assist the physics community in their machine learning efforts.\\n\",\n \"\\n\",\n \"**Contact**: [joeri.hermans@cern.ch](mailto:joeri.hermans@cern.ch)\\n\",\n \" [luca.canali@cern.ch](mailto:luca.canali@cern.ch)\\n\",\n \" [zbigniew.baranowski@cern.ch](mailto:zbigniew.baranowski@cern.ch)\\n\",\n \"\\n\",\n \"## Future work\\n\",\n \"\\n\",\n \"- Understanding \\\"theoretical\\\" meaning of a communication window.\\n\",\n \"- Apply compression for big weight updates when sending updates to the parameter server.\\n\",\n \"- Keep track of a gradient residual (this will reduce the bandwidth due to sparsity).\\n\",\n \"- Evaluation of algorithms with GPU's.\\n\",\n \"- Optimization of parameter sharing (e.g., sockets instead of REST API).\\n\",\n \"- Use \\\"famous\\\" ConvNet architectures with well known datasets in order to have a more sound evaluation.\\n\",\n \"- Add threaded queue to process asynchronous updates.\\n\",\n \"- Training accuracy while training.\\n\",\n \"- Stop on target loss.\\n\",\n \"\\n\",\n \"## Acknowledgments\\n\",\n \"\\n\",\n \"Many thanks to Zbigniew Baranowski and Luca Canali of the IT-DB group, and to Jean-Roch Vlimant, Maurizio Pierini, and Federico Presutti of the EP-UCM group for their collaboration on this work.\\n\",\n \"\\n\",\n \"## GitHub repository\\n\",\n \"\\n\",\n \"[https://github.com/JoeriHermans/dist-keras/](#https://github.com/JoeriHermans/dist-keras)\"\n ]\n }\n ],\n \"metadata\": {\n \"anaconda-cloud\": {},\n \"kernelspec\": {\n \"display_name\": \"Python 2\",\n \"language\": \"python\",\n \"name\": \"python2\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 2\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython2\",\n \"version\": \"2.7.13\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 0\n}\n"
},
{
"path": "mkdocs.yml",
"content": "# Project information\nsite_name: Distributed Keras\nsite_description: Distributed Deep Learning with Apache Spark and Keras.\nsite_author: Joeri Hermans\nsite_url: 'http://dist-keras.joerihermans.com'\n\n# Page definitions.\npages:\n- Home: index.md\n- Optimizers: optimizers.md\n- License: license.md\n# Documentation and theme configuration\ntheme: readthedocs\ndocs_dir: 'docs'\nmarkdown_extensions:\n- admonition\nextra:\n version: '0.1.0'\n palette:\n primary: 'grey'\n accent: 'light blue'\n author:\n github: 'JoeriHermans'\n twitter: 'joeri_hermans'\n\n# Copyright\ncopyright: 'Copyright (c) 2016 Joeri Hermans'\n\n# Repository\nrepo_name: 'GitHub'\nrepo_url: 'https://github.com/JoeriHermans/dist-keras'\n"
},
{
"path": "resources/blog-posts/css/main.css",
"content": "/**\n * joerihermans.com main stylesheet.\n *\n * @author Joeri Hermans\n * @version 0,1\n * @since 28 June 2016\n */\n\n/** BEGIN Imports. ************************************************************/\n\n@import url('https://fonts.googleapis.com/css?family=Roboto:300');\n\n/** END Imports. **************************************************************/\n\n/** BEGIN General. ************************************************************/\n\nhtml, body {\n background: white;\n color: #1b1b1b;\n font-family: 'Roboto', sans-serif;\n height: 100%;\n margin: 0;\n font-size: 0.95em;\n}\n\nbody {\n padding: 50px;\n}\n\na {\n color: #4078c0;\n cursor: pointer;\n text-decoration: none;\n}\n\na:hover {\n text-decoration: none;\n}\n\na img {\n border: 0;\n}\n\n.left {\n float: left !important;\n}\n\n.right {\n float: right !important;\n}\n\n.inlined-code {\n border-radius: 2px;\n background: #f0f0f0;\n padding: 2px 4px;\n}\n\n.center-text {\n text-align: center;\n}\n\n.image-container {\n text-align: center;\n}\n\n/** END General. **************************************************************/\n\n/** BEGIN Math. ***************************************************************/\n\n.equation-math {\n text-align: center;\n margin: 40px 0;\n position: relative;\n}\n\n.equation-math span {\n text-align: left;\n}\n\n.equation-math-number {\n position: absolute;\n top: 0;\n bottom: 0;\n margin: 0;\n height: 100%;\n line-height: 40px;\n right: 20px;\n font-size: 0.9em;\n color: #1b1b1b;\n font-weight: bold;\n}\n\n/** END Math. *****************************************************************/\n\n/** BEGIN Blog. ***************************************************************/\n\n.blog-figure-container {\n text-align: center;\n}\n\n.blog-figure-container img {\n max-width: 100%;\n}\n\n.blog-figure-container p {\n text-align: left;\n}\n\n/** END Blog. ****************************************************************/\n\n/** BEGIN Highlight. *********************************************************/\n\n.hljs {\n display: block;\n overflow-x: auto;\n padding: 10px 30px;\n background: #f0f0f0;\n}\n\n.hljs,\n.hljs-subst {\n color: #444;\n}\n\n.hljs-comment {\n color: #888888;\n}\n\n.hljs-keyword,\n.hljs-attribute,\n.hljs-selector-tag,\n.hljs-meta-keyword,\n.hljs-doctag,\n.hljs-name {\n font-weight: bold;\n}\n\n.hljs-type,\n.hljs-string,\n.hljs-number,\n.hljs-selector-id,\n.hljs-selector-class,\n.hljs-quote,\n.hljs-template-tag,\n.hljs-deletion {\n color: #d4645c;\n}\n\n.hljs-title,\n.hljs-section {\n color: #d4645c;\n font-weight: bold;\n}\n\n.hljs-literal {\n color: #78A960;\n}\n\n.hljs-built_in,\n.hljs-bullet,\n.hljs-code,\n.hljs-addition {\n color: #397300;\n}\n\n.hljs-meta {\n color: #1f7199;\n}\n\n.hljs-meta-string {\n color: #4d99bf;\n}\n\n.hljs-emphasis {\n font-style: italic;\n}\n\n.hljs-strong {\n font-weight: bold;\n}\n\n/** END Highlight. ***********************************************************/\n"
},
{
"path": "resources/blog-posts/js/highlight.pack.js",
"content": "/*! highlight.js v9.5.0 | BSD3 License | git.io/hljslicense */\n!function(e){var n=\"object\"==typeof window&&window||\"object\"==typeof self&&self;\"undefined\"!=typeof exports?e(exports):n&&(n.hljs=e({}),\"function\"==typeof define&&define.amd&&define([],function(){return n.hljs}))}(function(e){function n(e){return e.replace(/[&<>]/gm,function(e){return I[e]})}function t(e){return e.nodeName.toLowerCase()}function r(e,n){var t=e&&e.exec(n);return t&&0===t.index}function a(e){return k.test(e)}function i(e){var n,t,r,i,o=e.className+\" \";if(o+=e.parentNode?e.parentNode.className:\"\",t=B.exec(o))return R(t[1])?t[1]:\"no-highlight\";for(o=o.split(/\\s+/),n=0,r=o.length;r>n;n++)if(i=o[n],a(i)||R(i))return i}function o(e,n){var t,r={};for(t in e)r[t]=e[t];if(n)for(t in n)r[t]=n[t];return r}function u(e){var n=[];return function r(e,a){for(var i=e.firstChild;i;i=i.nextSibling)3===i.nodeType?a+=i.nodeValue.length:1===i.nodeType&&(n.push({event:\"start\",offset:a,node:i}),a=r(i,a),t(i).match(/br|hr|img|input/)||n.push({event:\"stop\",offset:a,node:i}));return a}(e,0),n}function c(e,r,a){function i(){return e.length&&r.length?e[0].offset!==r[0].offset?e[0].offset\":y.tabReplace?n.replace(/\\t/g,y.tabReplace):void 0}):e}function h(e,n,t){var r=n?L[n]:t,a=[e.trim()];return e.match(/\\bhljs\\b/)||a.push(\"hljs\"),-1===e.indexOf(r)&&a.push(r),a.join(\" \").trim()}function p(e){var n,t,r,o,s,p=i(e);a(p)||(y.useBR?(n=document.createElementNS(\"http://www.w3.org/1999/xhtml\",\"div\"),n.innerHTML=e.innerHTML.replace(/\\n/g,\"\").replace(/
In the following blog posts we study the topic of Distributed Deep Learning, or rather, how to parallelize gradient descent using data parallel methods. We start by laying out the theory, while supplying you with some intuition into the techniques we applied. At the end of this blog post, we conduct some experiments to evaluate how different optimization schemes perform in identical situations. We also introduce dist-keras, which is our distributed deep learning framework built on top of Apache Spark and Keras. For this, we provide several notebooks and examples. This framework is mainly used to test our distributed optimization schemes, however, it also has several practical applications at CERN, not only because of the distributed learning, but also for model serving purposes. For example, we provide several examples which show you how to integrate this framework with Spark Streaming and Apache Kafka. Finally, these series will contain parts of my master-thesis research. As a result, they will mainly show my research progress. However, some might find some of the approaches I present here useful to apply in their own work.
\nIntroduction
\nUnsupervised feature learning and deep learning has shown that being able to train large models on vasts amount of data can drastically improve model performance. However, consider the problem of training a deep network with millions, or even billions of parameters. How do we achieve this without waiting for days, or even multiple weeks? Dean et al. [2] propose a different training paradigm which allows us to train and serve a model on multiple physical machines. The authors propose two novel methodologies to accomplish this, namely, model parallelism and data parallelism. In the following blog post, we briefly mention model parallelism since we will mainly focus on data parallel approaches.
\nSidenote: In order to simplify the figures, and make them more intuitive, we negate the gradient \\nabla f without adding a - sign in front. Thus, all gradient symbols in the following figures will be negated by default, unless stated otherwise. I actually forgot to negate the gradients in the figures, so mentioning this is rather an easy fix. However, this will be corrected in the final version of the master thesis.
\nModel parallelism
\nIn model parallelism, a single model is distributed over multiple machines. The performance benefits of distributing a deep network across multiple machines mainly depends on the structure of the model. Models with a large number of parameters typically benefit from access to more CPU cores and memory, thus, parallelizing a large model produces a significant performance increase, and thereby reducing the training time.
\nLet us start with a simple example in order to illustrate this concept more clearly. Imagine having a perceptron, as depicted in Figure 1. In order to parallelize this efficiently, we can view a neural network as a dependency graph, where the goal is to minimize the number of synchronization mechanisms, assuming we have unlimited resources. Furthermore, a synchronization mechanism is only required when a node has more than 1 variable dependency. A variable dependency is a dependency which can change in time. For example, a bias would be a static dependency, because the value of a bias remains constant over time. In the case for the perceptron shown in Figure 1, the parallelization is quite straightforward. The only synchronization mechanism which should be implemented resides in output neuron since y \\triangleq \\sigma(\\sum_i w_ix_i) where \\sigma is the activation function of the output neuron.
\n\n ![\"Model](\"img/model-parallelism.png\")
\n
\n \n Figure 1: A perceptron partitioned using the model parallelism paradigm. In this approach every input node is responsible for accepting the input x_i from some source, and multiplying the input with the associated weight w_i. After the multiplication, the result is sent to the node which is responsible for computing y. Of course, this node requires a synchronization mechanism to ensure that the result is consistent. The synchronization mechanism does this by waiting for the results y depends on.
\nData parallelism
\nData parallelism is an inherently different methodology of optimizing parameters. The general idea is to reduce the training time by having n workers optimizing a central model by processing n different shards (partitions) of the dataset in parallel. In this setting we distribute n model replicas over n processing nodes, i.e., every node (or process) holds one model replica. Then, the workers train their local replica using the assigned data shard. However, it is possible to coordinate the workers in such a way that, together, they will optimize a single objective. There are several approaches to achieve this, and these will be discussed in greater detail in the coming sections and blog posts.
\nNevertheless, a popular approach to optimize this objective is to employ a centralized parameter server. A parameter server is responsible for the aggregation of model updates, and parameter requests coming from different workers. The distributed learning process starts by partitioning a dataset into n shards. Every individual shard will be assigned to a particular worker. Next, a worker will sample mini-batches from its shard in order to train the local model replica. After every mini-batch (or multiple mini-batches), the workers will communicate a variable with the parameter server. This variable is in most implementations the gradient \\nabla f_i(x). Finally, the parameter server will integrate this variable by applying a specific update procedure which knows how to handle this variable. This process repeats itself until all workers have sampled all mini-batches from their shard. This high-level description is summarized in Figure 2.
\n\n ![\"Data](\"img/data-parallelism.png\")
\n
\n \n Figure 2: Schematic representation of a data parallel approach. In this methodology we spawn n workers (not necessarily on different machines), and assign a data shard (partition) of the dataset to every worker. Using this data shard, a worker i will iterate through all mini-batches to produce a gradient, \\nabla f_i(x) for every mini-batch x. Next, \\nabla f_i(x) is send to the parameter server, which will incorperate the gradient using an update mechanism.
\nApproaches
\nIn this section we discuss several approaches towards parallelizing gradient descent (GD). This is not an intuitive task since gradient descent is an inherently sequential algorithm where every data point (instance) provides a direction to a minimum. However, training a model with a lot of parameters while using a very large dataset, will result in a high training time. If one would like the reduce the training time, the obvious choice would be to buy better, or rather, more suitable hardware (e.g., a GPU). However, this is not always possible. For this reason, several attempts have been made to parallelize gradient descent. In the following subsections, we will examine some of the popular approaches to parallelize gradient descent, and provide some intuition into these techniques work, and how they should be used.
\nSynchronous Data Parallel Methods
\nThere are two distinct approaches towards solving data parallelism. Personally, the most conceptually straightforward one is synchronous data parallelism. In synchronous data parallelism, as depicted in Figure 3, all workers compute their gradients based on the same center variable. This means whenever a worker is done computing a gradient for the current batch, it will commit a parameter (i.e., the gradient or the parameterization of the model) to the parameter server. However, before incorporating this information into the center variable, the parameter server stores all the information until all workers have committed their work. After this, the parameter server will apply a specific update mechanism (depending on the algorithm) to incorporate the commits into the parameter server. In essence, one can see synchronous data parallelism as a way to parallelize the computation of a mini-batch.
\n\n ![\"Synchronous](\"img/synchronous-data-parallelism.png\")
\n
\n \n Figure 3: In a synchronous data parallel setting, one has n workers (not necessarily on different machines). At the start of the training procedure, every worker fetches the most recent center variable. Next, every worker will start their training procedure. After the computation of the gradient, a worker commits the computed information (gradient or parametrization, depending on the algorithm) to the parameter server. However, due to unmodeled system behaviour, some workers might induce a significant delay, which results in other workers to be taskless while still consuming the same memory resources.
\nHowever, due to unmodeled system behaviour of the workers, workers might commit their results with a certain delay. Depending on the system load, this delay can be quite significant. As a result, this data parallel method is a case of the age-old saying \"a synchronous data parallel method is only as strong, as the weakest worker in the cluster\" :-).
\nModel Averaging
\nIn essence, this is a data parallel approach as mentioned in the introduction. However, in contrary to more conventional data parallel approaches, there is no parameter server. In model averaging, every worker will get a copy of the model at the start of the training period. However, one can have different weight initialization techniques for the workers to cover more of the parameter space after several iterations, as shown in Figure 4. Though, it is not recommended to do this since this will result in very different solutions for every worker. Thus wasting initial iterations converging to a \"good solution\" on which all workers \"agree\". However, this problem is related to most distributed optimization algorithms discussed here, and will be discussed in more detail in the following blog posts.
\nAfter every worker is initialized with a copy of the model, all workers start the training procedure independently from each other. This means that during the training procedure, no communication between the workers occurs. Thus, eliminating the communication overhead that is present in approaches with parameter servers. After the end of an epoch, i.e., a full iteration of the dataset, the models are aggregated and averaged on a single worker. The resulting averaged model will then be distributed to all workers, where the training process repeats until the averaged model converges.
\n\n ![\"Model](\"img/model-averaging.png\")
\n
\n \n Figure 4: In this setting we have 4 independent workers, each having a randomly initialized model. In order to simplify the situation, let us assume we can obtain the gradient directly from E(\\theta), which is our loss function. In model averaging, every worker only applies gradient descent to its own model without communicating with other workers. After the end of an epoch, as shown in the center plot, the models are averaged in order to produce a central model. In the following epoch, the central model will be used as a starting point for all workers.
\nEASGD
\nEASGD, or Elastic Averaging SGD, introduced by Zhang et al. [3], is a distributed optimization scheme designed to reduce communication overhead with the parameter server. This is in contrast to approaches such as DOWNPOUR, which most of the time require a small communication window in order to converge properly. The issue with a small communication window is that the learning process needs to be stopped in order to synchronize the model with the parameter server, and as a result, limiting the throughput of the training process. Of course, the number of parameters in a model is also an important factor. For example, one can imagine that having a model with 100 MB worth of parameters could severely influence the training performance if every 5 mini-batches a synchronization with the parameter server would occur. Furthermore, the authors state that due to the distributed nature, exploration of the nearby parameter space by the workers actually improves the statistical performance of a model with respect to sequential gradient descent. However, at the moment, we do not have any evidence to support this claim, nor to deny it. What we do observe, is that the statistical performance of a model after a single epoch, is usually (significantly) less than a single epoch of Adam (sequential training) and ADAG (distributed training). However, if we would let EASGD reach the same amount of wallclock training time, then we still have an identical or slightly worse model performance. So there is evidence to suggest that this claim is not completely true, at least, in the case of EASGD. This however, requires more investigation.
\nThe authors solve the communication constraint by applying an \"elastic force\" between the parameters of the workers and the center variable. Furthermore, due to the elasticity and reduction in communication with the parameter server, the workers are allowed to explore the surrounding parameter space. As stated above, the authors claim that allowing for more exploration can be beneficial for the statistical performance of the model. However, we argue that, as in model averaging, this will only work well if the workers are in the neighbourhood of the center variable, we will show this empirically in the Experiments section. However, in contrast to model averaging, the workers are not synchronized with the center variable. This begs the question, how does EASGD ensure that the workers remain in the \"neighbourhood\" of the center variable? Because as in model averaging, too much exploration of the parameter space actually deteriorates the performance of the center variable, and may prevent convergence, because the workers cover inherently different spaces of the parameter space, as shown in Figure 4. However, if the elasticity parameter is too high, exploration will not take place at all.
\n\\theta^i_{t+1} = \\theta^i_t - \\eta\\nabla f(\\theta^i_t) - \\eta\\rho(\\theta^i_t - \\theta^c_t)
\n \\theta^c_{t+1} = (1 - n\\eta\\rho) \\theta^c_t + \\eta\\rho(\\frac{1}{n} \\sum_{i = 0}^{n} \\theta^i_t)
\n To fully understand the implications of the EASGD equations, as shown in Equation 1 and Equation 2, we refer the reader to Figure 5, which shows the intuition behind the elastic force. Having two vectors, the gradient \\nabla f, and the elastic difference \\eta\\rho(\\theta^i_t - \\theta^c_t) where \\eta is the learning rate and \\rho is the elasticity parameter, the authors say when \\rho is small, you allow for more exploration of the parameter space. This can be observed from Figure 5. When \\rho is small, the vector \\eta\\rho(\\theta^i_t - \\theta^c_t) will be small as well (unless the distance between the worker and the center variable is large). As a result, the attraction between the center variable and the worker is small, thus, allowing for more exploration of the parameter space.
\nAnalogously, imagine that you are walking with your dog, and the dog is responsible for getting you home (guiding you to a minimum). If you would let your dog drift too far away from you (because you have a leash which is very flexible (small \\rho)). In the most extreme case, the dog will get home without you because your leash was simply too flexible. As a result, the dog could not pull you home. At this point you think, maybe I should buy more dogs? Thinking that together they will help you. However, due to the nature of these creatures you soon realize that instead of going home, they all go to different places (multiple workers in the parameter space having different inputs, e.g., one dog sees a particular tree, while an other dog sees a bush, etc.). From this experience, you notice that the problem is the leash, it is way too flexible because the dogs are all over the place. As a result, you buy a less flexible leash, with the effect that the dogs stay closer to you, and eventually \"pull\" together to bring you home faster.
\n\n ![\"EASGD\"](\"img/easgd.png\")
\n
\n \n Figure 5: The worker variable w is exploring the parameter space in order to optimize C. However, the amount of exploration is proportional by the elasticity factor \\rho, and the difference (\\theta^w_t - \\theta^C_t). In general, when \\rho is small, you allow for more exploration to occur. It is to be noted, that as in model averaging, too much exploration will actually deteriorate the statistical performance of a model (as shown in the first subfigure of Figure 4), because the workers do not agree on a good solution. Especially when you take into account that the center variable is updated using an average of the worker variables, shown in Equation 2.
\nNow, from Equation 1 and the intuition above, we can expect that for some worker update i within a communication window, the accumulated gradient is larger or equal to the elastic force. As a result, this prevents the workers from further exploration (as expected). However, a significant side-effect is that the following gradient computations are wasted since they are countered by the elastic difference, as shown in Figure 5. Using the analogy from above, this is equivalent to a situation where no matter how hard a dog is trying to pull, you just don't let it go any further. Thus, the efforts of the dog are wasted. This condition is described by Equation 3.
\n\n - \\sum_i - \\eta\\nabla f(x_{t + i};\\theta_{t + i}) \\geq -\\eta\\rho(\\theta_{t + i + 1} - \\theta_t^C)\n
\n A straightforward technique to prevent the squandering of computations after the condition described by Equation 3, is to simply check for this condition after the computation of every mini-batch. When this condition is met, then the term \\sum_i - \\eta\\nabla f(x_{t + i};\\theta_{t + i}) is communicated with the parameter server. As a result, we do not waste any computations, and furthermore, we loose a hyperparameter since the communication window is now controlled (indirectly) by the hyperparameter \\rho, which controls the elastic force. In essence, the core idea of ADAG (which will be mentioned later in this blog post), can also be applied to this scheme to even further improve the quality of the gradient updates, and making the optimization scheme less sensitive to other hyperparameters, e.g., the number of parallel workers.
\nAsynchronous Data Parallel Methods
\nIn order to overcome the significant delays induced by loaded workers in synchronous data parallelism, and thereby decrease the training time even further, let us simply remove the synchronization constraint. However, this imposes several other effects, and some of them are not very obvious. The conceptually simplest, is parameter staleness. Parameter staleness is simply the number of commits other workers performed between the last pull (center variable synchronization), and the last commit (parameter update) of the current worker. Intuitively, this implies that a worker is updating a \"newer\" model using gradients based on a previous parametrization of that model. This is shown in Figure 6.
\n\n ![\"Asynchronous](\"img/asynchronous-data-parallelism.png\")
\n
\n \n Figure 6: In asynchronous data parallelism, training time is even further reduced (on average) due to the removal of the synchronization mechanism in synchronous data parallelism. However, this induces several effects such as parameter staleness, and asynchrony induced momentum.
\nNote: It is not required to read the paragraphs below, unless you really want to. However, the take-away point is: increasing the number of parallel workers behaves like adding more momentum.
\nThe other, less intuitive side-effect is asynchrony induced momentum [1]. Roughly stated, this means that adding more workers to the problem also adds more implicit momentum to the optimization process. This implicit momentum is the result of the queuing model required by asynchrony. Note that some approaches, such as Hogwild! do not require locking mechanisms, since they assume sparse gradient updates. However, distributed SGD works with dense gradient updates as well. We also confirm the statements of the authors that adding more asynchronous workers to the problem actually deteriorates the statistical performance of the model when using algorithms which do not take staleness and asynchrony into account. Furthermore, they state that the behaviour of an asynchronous algorithm is roughly described by Equation 4. Which implies that the implicit momentum produced by asynchrony is (1 - \\frac{1}{n}).
\nE[\\theta_{t+1} - \\theta_t] = (1 - \\frac{1}{n})E[\\theta_t - \\theta_{t-1}] - \\frac{1}{n}\\eta E \\nabla f_i(\\theta_t)
\n But personally, I think this is not the complete story. I agree with the nicely formalized queueing model, and that in general, an increase in the number of asynchronous workers decreases the statistical performance of a model (we also observe this in our experiments). However, I would say that the effect rather behaves like momentum, but cannot be necessarily be defined as such (with ADAG, we do not observe this effect, at least for 30 parallel processes). We will go more in-depth into this topic in the following blog posts, since this is still a topic that requires some more research on my part.
\nAsynchronous EASGD
\nThe update scheme of asynchronous EASGD is quite similar, however, there are some important details. In the following paragraphs we will call the vector - \\eta\\rho(\\theta^i_t - \\theta^c_t) the elastic difference, and thereby following the notation of the paper. Remember that in the synchronous version this vector is actually used to enforce the exploration policy. Meaning, in Equation 1 this vector has the task to not let a worker drift too \"far\" from the center variable. Repeating the analogy with the dogs, imagine having a dog with an elastic leash. The further the dog walks away from you (the center variable), the stronger the force will be to pull it back. As a result, at some point the force the dogs exerts will be equal to the force the elastic leash exerts in the opposite direction. At this point, the dog cannot move any further. This is exactly what happens when the elastic difference is applied to a worker, as shown in Figure 5.
\nIn the asynchronous version, the elastic difference has the same function. However, it will also be used to update the center variable. As stated in the paragraph above, the elastic difference is actually used to limit exploration. However, if we negate the elastic difference, which is + \\eta\\rho(\\theta^i_t - \\theta^c_t), then the elastic difference can be used to optimize the center variable (reverse the arrow in Figure 5), while still holding true to the communication constraints EASGD is trying to solve.
\nDOWNPOUR
\nIn DOWNPOUR, whenever a worker computes a gradient (or a sequence of gradients), the gradient is communicated with the parameter server. When the parameter server receives the gradient update from a worker, it will incorporate the update in the center variable, as shown in Figure 7. Contrary to EASGD, DOWNPOUR does not assume any communication constraints. Even more, if frequent communication with the parameter server does not take place (in order to reduce worker variance), DOWNPOUR will not converge (this is also related to the asynchrony induces momentum issue, see Figure 8). This is because of the same issues discussed in the sections above. If we allow the workers to explore \"too much\" of the parameter space, then the workers will not work together on finding a good solution for the center variable. Furthermore, DOWNPOUR does not have any intrinsic mechanisms in place to remain in the neighbourhood of the center variable. As a result, if you would increase the communication window, you would proportionally increase the length of the gradient which is sent to the parameter server, thus, the center variable is updated more aggressively in order to keep the variance of the workers in the parameter space \"small\".
\n\n ![\"DOWNPOUR](\"img/downpour.gif\")
\n
\n \n Figure 7: Animation of DOWNPOUR with 20 parallel workers (blue) with identical learning rates which are trying to optimize a single objective (center variable, red) compared to regular sequential gradient descent (green). From this animation we can observe the momentum induced by the asynchrony of the parallel workers, as discussed above.
\n\n ![\"DOWNPOUR](\"img/downpour_momentum.gif\")
\n
\n \n Figure 8: Animation of DOWNPOUR with 40 parallel workers. In this case, the implicit momentum produced by the number of workers causes the algorithm to diverge.
\nADAG
\nWe noticed that a large communication window is correlated with a decline in model performance. Using some simulations (like DOWNPOUR, as shown above), we noticed that this effect can be mitigated when you normalize the accumulated gradient with the communication window. This has several positive effects, for one, you are not normalizing with respect to the number of parallel workers, thus, as a result, you are not losing the (convergence speed) benefit of parallelizing gradient descent. This has as a side-effect, that the variance of the workers with respect to the center variable will also remain small, thus contributing positively to the central objective! Furthermore, because of the normalization, you are less sensitive to hyperparametrization, especially regarding the communication window. However, it is to say that a large communication window typically also degrades the performance of the model because you allow the workers to explore more of the parameter space using the samples from their data shard. In our first prototype, we adapted DOWNPOUR to fit this idea. We observed the following results. First, we observe a significant increase in model performance, even compared to a sequential optimization scheme such as Adam. Second, compared to DOWNPOUR, we can increase the communication window with a factor 3. Thus, allow to utilize the CPU resources more efficiently, and decreasing the total training time even further. Finally, normalizing the accumulated gradient allows us the increase the communication window. As a result, we are able to match the training time of EASGD, and achieve the roughly the same (sometimes better, sometimes worse).
\n\n \\Large \\frac{\\sum_{i=0}^{\\tau}-\\eta\\nabla f(x_{t + i};\\theta_{t + i})}{\\tau}\n
\n To conclude, the core idea of ADAG, or asynchronous distributed adaptive gradients, can be applied to any distributed optimization scheme. Using our observations, and intuition (especially with respect to implicit momentum due to asynchrony), we can make a calculated guess that the idea of normalized accumulated gradients can be applied to any distributed optimization scheme. However, we need to conduct several experiments in order to verify this claim. ADAG will be discussed in detail in the following blog posts.
\nDistributed Keras
\nDistributed Keras is a distributed deep learning framework built on top of Apache Spark and Keras with the goal to significantly reduce the training using distributed machine learning algorithms, and allow bigger than memory datasets. This project initially started as a prototype with the CMS collaboration. However, the project has seen several iterations already since its start in August 2016.
\nArchitecture
\nIn essence, a training procedure is passed on to the Spark workers as a lambda function. However, in order to pass on multiple parameters, e.g., port number of the parameter server, we wrap everything in an object, and define a train function which accepts the parameters required by Spark. To give a complete overview, let us assume that a user just called the train method shown in the code block below. The trainer object will first allocate and start a parameter server on the Spark driver. Next, it allocates the worker procedure, which holds all parameters and procedures to train the Keras model. Furthermore, in order to comply with the required number of parallel workers, we partition the dataset according to this specific amount. However, when processing big datasets it is recommended to increase the parallelism factor, this will prevent that some workers remain idle while other (slower) workers are still processing their old batch (in literature, this is know as the straggler problem). In such cases, we recommend a parallelism factor of 3, as suggested by the Spark documentation.
\nAlthough, one needs to consider the implications of a large parallelism factor. Basically, the parallelism is proportional to the number of partitions you will create. So lets say you assign 20 workers to a specific task, with a parallelism factor of 3. Spark will then repartition the dataset into 60 shards. Now, before a worker starts processing a partition, it first has to load all the Python libraries which are required to process the task, next, it will also have to deserialize and compile the Keras model. This induces a significant overhead. So this technique is only effective on non-heterogenous systems (meaning, different hardware, or variable load per worker), and large datasets due to the large \"warmup\" overhead.
\n\n ![\"Distributed](\"img/distkeras-architecture.png\")
\n
\n \n Figure 9: Imagine we have a Spark Context with k executors, and l cores per executor. Using these parameters, there will be n = k \\cdot l workers allocated by dist-keras. However, if you would like to use a smaller amount of parallel workers, you can simply parameterize the training algorithm without having to reinitialize the Spark Context.
\n\n \n # Allocate the parameter server.\n self.parameter_server = self.allocate_parameter_server()\n # Start the communication service.\n self.start_service()\n # Allocate a worker.\n worker = self.allocate_worker()\n # Repartition in order to fit the number of workers.\n num_partitions = dataframe.rdd.getNumPartitions()\n # Assign the dataset.\n dataset = dataframe\n if shuffle:\n dataset = shuffle(dataset)\n if num_partitions > self.num_workers:\n dataset = dataset.coalesce(self.num_workers)\n else:\n dataset = dataset.repartition(self.num_workers)\n dataset.cache()\n # Iterate through the epochs (some trainers require a result).\n dataset.rdd.mapPartitionsWithIndex(worker.train).collect()\n # Stop the communication service.\n self.stop_service()\n\n return self.parameter_server.get_model()\n \n \n Experiments
\nIn the following experiments we set up the different optimization schemes against each other, i.e. (sequential) Adam, (distributed) DOWNPOUR, Asynchronous EASGD, and ADAG, and evaluate them using the MNIST dataset (samples are shown in Figure 10). We will use the following parameters during our experiments:
\n- \n
- Multilayer perceptron with 1 000 000 trainable parameters (~4 MB model) (complete model summarized below) \n
- 4 sample mini-batches \n
- 1 epoch \n
- Parallelism factor: 1 \n
- Adam as worker optimizer \n
- Communication windows:\n
- \n
- DOWNPOUR: 5 \n
- ADAG: 5 \n
- Asynchronous EASGD: 32 \n
\n - 20 parallel workers:\n
- \n
- 10 compute nodes with 10 Gbps network cards \n
- 2 processes per compute node (32 cores) \n
\n
\n \nmlp = Sequential()\nmlp.add(Dense(1000, input_shape=(784,)))\nmlp.add(Activation('relu'))\nmlp.add(Dropout(0.2))\nmlp.add(Dense(200))\nmlp.add(Activation('relu'))\nmlp.add(Dropout(0.2))\nmlp.add(Dense(10))\nmlp.add(Activation('softmax'))\n \n \n \n ![\"MNIST](\"img/mnist.png\")
\n
\n \n Figure 10: The MNIST dataset is a collection of handwritten digits. This dataset is usually used as a \"unit test\" for optimization algorithms. Every sample consists of 784 pixels, with values ranging between 0 and 255. We normalize these using our framework dist-keras, which is built on top of Apache Spark, thus, profiting from the parallelization.
\nIn the following experiments we evaluate the accuracy of the central variable, and the training time (wallclock) compared to the number of parallel workers. Although this is a relatively small dataset, it gives us some indications into the scaling abilities of the optimization schems. In the following blog posts we will also focus on large scale deep learning, meaning, we will handle very large datasets and train models in a data parallel setting.
\nDOWNPOUR
\n\n ![\"DOWNPOUR](\"img/experiment_downpour.png\")
\n
\n \n Figure 11: A key observation in this experiment is that DOWNPOUR actually diverges when it reaches a critical amount of implicit momentum, as shown in Figure 8. We made this observation in several other datasets as well. However, the constantly declining performance the authors in [1] is not observed. Rather, we have a sudden decline in model performance. This is rather contradictory to the claims made in [1]. According to their theory, we should not see a sudden decline in model performance, but rather a steady decline. As a result, we think that their statement \"there exists a limit to asynchrony\" is false as well. Though, their intuition is correct! Furthermore, on the left, we see the scaling of the algorithm. We actually expected that the scaling should work better, however, this could be because of the unbalanced partitions (we are doing experiments with other partitioners to correct for this) and relatively small dataset.
\nAsynchronous EASGD
\n\n ![\"Asynchronous](\"img/experiment_easgd.png\")
\n
\n \n Figure 12: As stated above, EASGD is an algorithm designed with communication constraints in mind, which is a realistic constraint. As a result, the authors incorporate an elastic force which allows the worker to explore a certain area of the neighbouring parameter space w.r.t. the center variable. As a result, it will not have an immediate decline in model performance, as observed in DOWNPOUR, but rather a steady decline. This decline (with respect to the number of model performance), is due to the increased amount of staleness (since the center variable will have covered more distance because of the queuing model) compared to the worker. As a result, the positive information a worker can contribute is proportional to the elastic difference, and this elastic difference will be smaller when the number of parallel workers is higher (due to parameter staleness). However, since EASGD scales very well with the number of workers, we simply match the training time of ADAG or DOWNPOUR. However, even if we would match the training time, EASGD usually results in a lower accuracy compared to ADAG. This is phenomena is subject to further study, as it is not really completely understood why this is actually happening. Furthermore, it also consumes more CPU compared to ADAG, if we would match the model performance of ADAG (ADAG spends a lot of time waiting for network operations).
\nADAG
\n\n ![\"ADAG](\"img/experiment_adag.png\")
\n
\n \n Figure 13: If we would assume no communication constraints, then how would we solve the problem DOWNPOUR has? Averaging the gradients will work, but it is not very desireable since the gradient will act as if they were a sequential optimization algorithm. So what if we would normalize with respect to the communication window? Since this really is the parameter which induces parameter staleness, as can be observed from Figure 12 (declining model performance). An interesting observation we can make here is the absence of any decline in model performance (compared to DOWNPOUR and EASGD). We think this is because one of the following reasons; for one, we keep the variance of the workers small (limited exploration), and normalize the accumulated gradient on the workers with the communication window (which is a prime factor in implicit momentum).
\nInfluence of the communication window on accuracy and training time
\nIn the following set of experiments we will investigate the influence of the communication window on accuracy and training time. The communication window is a hyperparameter which defines the frequency of communication with the parameter server. A communication window of 35 means that a worker will accumulate 35 mini-batch updates, and finally synchronizes with the parameter server. In the experiments, all optimization schemes use identical hyperparameters, where the only variable between tests is the communication window. As before, we will use MNIST as a dataset, a mini-batch of size 4, and Adam as the worker optimizer.
\n\n ![\"Influence](\"img/experiment_communication_window_accuracy.png\")
\n
\n \n Figure 14: As expected, DOWNPOUR is not able to handle large communication windows. EASGD on the other hand is not able to handle small communication windows! As stated above, this is because the elastic force (due to the number of workers) is stronger then the exploration of the parameter space. Thus, causing EASGD to not converge. ADAGA on the other hand is able to handle the varying communication window, however, a slight decline in model performance is observed. This is expected due to the increase in exploration of the parameter space by the workers.
\n\n ![\"Influence](\"img/experiment_communication_window_training_time.png\")
\n
\n \n Figure 15: Again, the training time of all optimization schemes decrease significantly when the communication window is increased. However, we think we can further decrease the training time by allocated a thread in every worker which sole responsibility is to send the parameters to the parameter server. However, this is an idea that has yet to be explored. To conclude, we suggest to make a trade-off between training time and accuracy, in the case of ADAG, we recommend a communication window of 10-15, since this hyperparametrization achieves similar model performance. However, when applying this to a differen dataset. We recommend that you test these settings for yourself, since they can differ.
\nSummary
\nIn this work we gave the reader an introduction to the problem of distributed deep learning, and some of the aspects which one needs to consider when applying it, such as, for example, implicit momentum. We also suggested some techniques which are able to significantly improve existing distributed optimization schemes. Furthermore, we introduced our framework, dist-keras, and applied different distributed optimization schemes to the MNIST dataset. Finally, we also provided several production-ready examples and notebooks.\n
Acknowledgements
\nThis work was done as part of my Technical Student contract at CERN IT. I would like to thank Zbigniew Baranowski and Luca Canali of the IT-DB group, Volodimir Begy of the University of Vienna, and to Jean-Roch Vlimant, Maurizio Pierini, and Federico Presutti (CalTech) of the EP-UCM group for their collaboration on this work.
\nReferences
\n- \n
- Mitliagkas, I., Zhang, C., Hadjis, S., & Ré, C. (2016). Asynchrony begets Momentum, with an Application to Deep Learning. arXiv preprint arXiv:1605.09774. \n
- Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., ... & Ng, A. Y. (2012). Large scale distributed deep networks. In Advances in neural information processing systems (pp. 1223-1231). \n
- Zhang, S., Choromanska, A. E., & LeCun, Y. (2015). Deep learning with elastic averaging SGD. In Advances in Neural Information Processing Systems (pp. 685-693). \n
- The MNIST database, of handwritten digits. \n