\n ![\"Chart\"](\"img/Feedforwardnets-1.png\")
\n
\n\n \nval MNISTtrain : Dataset = Hype.Dataset\n X: 784 x 59000\n Y: 1 x 59000\nval MNISTvalid : Dataset = Hype.Dataset\n X: 784 x 1000\n Y: 1 x 1000\nval MNISTtest : Dataset = Hype.Dataset\n X: 784 x 10000\n Y: 1 x 10000\n\n\n*)\n\nMNISTtrain.[..5].VisualizeXColsAsImageGrid(28) |> printfn \"%s\"\n\n(**\n\n [lang=cs]\n Hype.Dataset\n X: 784 x 6\n Y: 1 x 6\n X's columns reshaped to (28 x 28), presented in a (2 x 3) grid:\n DM : 56 x 84\n \n \n \n ·▴█ \n ■■♦█· █■ \n ▪███■▪ -██■- \n ·■███♦● ·●██■▪ \n -♦▪ ·████♦ -♦█♦ \n -♦■▪· █■█● ■█· \n ·●██♦ ██●· ██· \n ·▴ ·●██▪ -■██▪ ■█▪ \n ·■▪ ·▪■■▪ ·███▪ ·▴· ♦█▪ \n ♦■▪ ·■■▴ ♦███●▴▴♦██● ██▴ \n █ ■█· ■█■█■██████■ ■█▪ \n ■█· - ██▴█████████▴ ·██· \n ●█▪ █▪●■████■███▴ ·███♦· \n ·■■ █▴●- · ●█· ●♦♦♦█▪ \n ■█· █●■ ▪█ ▴█- \n ·█● ███· ·●■■██ ■█ \n ·█● █●■▴▴██████● ██ \n ●█▴ ♦█████████▴ ▴█▪ \n -♦██♦ ▪█████♦▪· ● █■ \n ▴■■▴ ▪▪▪ █ ·■♦ \n ■♦▪♦█▪ \n \n \n \n \n \n \n \n \n \n \n ·■ ·♦ ▪■♦- ·●●█■· \n ██· ♦█● ██■■█▪-●· ▪■·▴● \n ██▴ ♦██▪ ■█- ▴███■ █♦ \n -██ ♦██● ▪█· ▪██▪ ▪█♦ ■▴ \n -██ ♦██■ █● ♦██▴ ♦█- ██■ \n ▴█♦ ▴██■ ▴█ ·██■· ♦█- ███ \n ♦█♦ -███ ▴█·▴██♦ ▪██■███· \n ♦█· ·██● ████▴ ■██●■█· \n ♦█▴ ··-███● ▪██♦ · ♦█· \n ♦██■■■■██████● ·■███ ██ \n ▪█████████■♦█● ♦██■●█ █■ \n ●██████▴- █■ ▴██■- ●█ ▴█▪ \n ·███♦· ██ ♦█■▴ █● ██▴ \n ▪▪ ██▪ ███ ■█· ██▴ \n ●██ ●████■■██- ██ \n ▴██ -▪▪▪▪▪▪▪- ██ \n ██ -██ \n ■█ -█● \n ●█· -█▪ \n ·█ █ \n \n \n\n### Defining the model\n\nWe define a neural network with 3 layers: (1) a hidden layer with 300 units, followed by ReLU activation, (2) a hidden layer with 100 units, followed by ReLU activation, (3) a final layer with 10 units, followed by softmax transformation.\n*)\n\nlet n = FeedForward()\nn.Add(Linear(28 * 28, 300, Initializer.InitReLU))\nn.Add(reLU)\nn.Add(Linear(300, 100, Initializer.InitReLU))\nn.Add(reLU)\nn.Add(Linear(100, 10))\nn.Add(fun m -> m |> DM.mapCols softmax) // Note the free inline implementation of the layer\n\nn.ToString() |> printfn \"%s\"\n\n(**\n [lang=cs]\n Hype.Neural.FeedForward\n Learnable parameters: 266610\n (0) -> (1) -> (2) -> (3) -> (4) -> (5)\n\n (0): Hype.Neural.Linear\n 784 -> 300\n Learnable parameters: 235500\n Init: ReLU\n W : 300 x 784\n b : 300\n\n (1): Hype.Neural.Activation\n\n (2): Hype.Neural.Linear\n 300 -> 100\n Learnable parameters: 30100\n Init: ReLU\n W : 100 x 300\n b : 100\n\n (3): Hype.Neural.Activation\n\n (4): Hype.Neural.Linear\n 100 -> 10\n Learnable parameters: 1010\n Init: Standard\n W : 10 x 100\n b : 10\n\n (5): Hype.Neural.Activation\n*)\n\n\n(**\n\n### Freely implementing transformation layers\n\nNow let's have a closer look at how we implemented the nonlinear transformations between the linear layers. \n\nYou might think that the instances of **reLU** in **n.Add(reLU)** above refer to a particular layer structure previously implemented as a layer module within the library. They don't. **reLU** is just a matrix-to-matrix elementwise function.\n\n**An important thing to note** here is that the activation/transformation layers added with, for example, **n.Add(reLU)**, can be **any matrix-to-matrix function that you can express in the language,** unlike commonly seen in many machine learning frameworks where you are asked to select a particular layer type that has been implemented beforehand with it's (1) forward evaluation code and (2) reverse gradient code w.r.t. layer inputs, and (3) reverse gradient code w.r.t. any layer parameters. In such a setting, a new layer design would require you to add a new layer type to the system and carefully implement these components.\n\nHere, because the system is based on nested AD, you can freely use any matrix-to-matrix transformation as a layer, and the forward and/or reverse AD operations of your code will be handled automatically by the underlying system. For example, you can write a layer like this: \n*)\n\nn.Add(fun w ->\n let min = DM.Min(w)\n let range = DM.Max(w) - min\n (w - min) / range)\n\n(** \nwhich will be a normalization layer, scaling the values to be between 0 and 1.\n\nIn the above model, this is how the softmax layer is implemented as a mapping of the vector-to-vector **softmax** function to the columns of a matrix. \n\n*)\n\nn.Add(fun m -> m |> DM.mapCols softmax) \n\n(**\nIn this particular example, the output matrix has 10 rows (for the 10 target classes) and each column (a vector of size 10) is individually passed through the **softmax** function. The output matrix would have as many columns as the input matrix, representing the class probabilities of each input.\n*)\n\n\n(**\n### Weight initialization schemes\n\nWhen layers with learnable weights are created, the weights are initialized using one of the following schemes. The correct initialization would depend on the activation function immediately following the layer and would take the fan-in/fan-out of the layer into account. If a specific scheme is not specified, the **InitStandard** scheme is used by default. These implementations are based on existing machine learning literature, such as _\"Glorot, Xavier, and Yoshua Bengio. \"Understanding the difficulty of training deep feedforward neural networks.\" International conference on artificial intelligence and statistics. 2010\"_.\n\n*)\n\ntype Initializer =\n | InitUniform of D * D\n | InitNormal of D * D\n | InitRBM of D\n | InitReLU\n | InitSigmoid\n | InitTanh\n | InitStandard\n | InitCustom of (int->int->D)\n override i.ToString() =\n match i with\n | InitUniform(min, max) -> sprintf \"Uniform min=%A max=%A\" min max\n | InitNormal(mu, sigma) -> sprintf \"Normal mu=%A sigma=%A\" mu sigma\n | InitRBM sigma -> sprintf \"RBM sigma=%A\" sigma\n | InitReLU -> \"ReLU\"\n | InitSigmoid -> \"Sigmoid\"\n | InitTanh -> \"Tanh\"\n | InitStandard -> \"Standard\"\n | InitCustom f -> \"Custom\"\n member i.InitDM(m, n) =\n let fanOut, fanIn = m, n\n match i with\n | InitUniform(min, max) -> Rnd.UniformDM(m, n, min, max)\n | InitNormal(mu, sigma) -> Rnd.NormalDM(m, n, mu, sigma)\n | InitRBM sigma -> Rnd.NormalDM(m, n, D 0.f, sigma)\n | InitReLU -> Rnd.NormalDM(m, n, D 0.f, sqrt (D 2.f / (float32 fanIn)))\n | InitSigmoid -> let r = D 4.f * sqrt (D 6.f / (fanIn + fanOut)) in Rnd.UniformDM(m, n, -r, r)\n | InitTanh -> let r = sqrt (D 6.f / (fanIn + fanOut)) in Rnd.UniformDM(m, n, -r, r)\n | InitStandard -> let r = (D 1.f) / sqrt (float32 fanIn) in Rnd.UniformDM(m, n, -r, r)\n | InitCustom f -> DM.init m n (fun _ _ -> f fanIn fanOut)\n member i.InitDM(m:DM) = i.InitDM(m.Rows, m.Cols)\n\n(**\n### Training\n\nBefore training, let's visualize the weights of the first layer in a grid where each row of the weight matrix of the first layer is shown as a 28-by-28 image. It is an image of random weights, as expected.\n*)\n\nlet l = (n.[0] :?> Linear)\nl.VisualizeWRowsAsImageGrid(28) |> printfn \"%s\"\n\n(**\n
\nHype.Neural.Linear\n 784 -> 300\n Learnable parameters: 235500\n Init: ReLU\n W's rows reshaped to (28 x 28), presented in a (17 x 18) grid:\n\n\n
\n \n[12/11/2015 22:42:07] --- Training started\n[12/11/2015 22:42:07] Parameters : 266610\n[12/11/2015 22:42:07] Iterations : 1180\n[12/11/2015 22:42:07] Epochs : 2\n[12/11/2015 22:42:07] Batches : Minibatches of 100 (590 per epoch)\n[12/11/2015 22:42:07] Training data : 59000\n[12/11/2015 22:42:07] Validation data: 1000\n[12/11/2015 22:42:07] Valid. interval: 10\n[12/11/2015 22:42:07] Method : Gradient descent\n[12/11/2015 22:42:07] Learning rate : RMSProp a0 = D 0.00100000005f, k = D 0.899999976f\n[12/11/2015 22:42:07] Momentum : Nesterov D 0.899999976f\n[12/11/2015 22:42:07] Loss : Cross entropy after softmax layer\n[12/11/2015 22:42:07] Regularizer : L2 lambda = D 9.99999975e-05f\n[12/11/2015 22:42:07] Gradient clip. : None\n[12/11/2015 22:42:07] Early stopping : Stagnation thresh. = 400, overfit. thresh. = 100\n[12/11/2015 22:42:07] Improv. thresh.: D 0.995000005f\n[12/11/2015 22:42:07] Return best : true\n[12/11/2015 22:42:07] 1/2 | Batch 1/590 | D 2.383214e+000 [- ] | Valid D 2.411374e+000 [- ] | Stag: 0 Ovfit: 0\n[12/11/2015 22:42:08] 1/2 | Batch 11/590 | D 6.371681e-001 [↓▼] | Valid D 6.128169e-001 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 22:42:08] 1/2 | Batch 21/590 | D 4.729548e-001 [↓▼] | Valid D 4.779414e-001 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 22:42:09] 1/2 | Batch 31/590 | D 4.792733e-001 [↑ ] | Valid D 3.651254e-001 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 22:42:10] 1/2 | Batch 41/590 | D 2.977416e-001 [↓▼] | Valid D 3.680202e-001 [↑ ] | Stag: 10 Ovfit: 0\n[12/11/2015 22:42:10] 1/2 | Batch 51/590 | D 4.242567e-001 [↑ ] | Valid D 3.525212e-001 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 22:42:11] 1/2 | Batch 61/590 | D 2.464822e-001 [↓▼] | Valid D 3.365663e-001 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 22:42:11] 1/2 | Batch 71/590 | D 6.299557e-001 [↑ ] | Valid D 3.981607e-001 [↑ ] | Stag: 10 Ovfit: 0\n...\n[12/11/2015 22:43:21] 2/2 | Batch 521/590 | D 1.163270e-001 [↓ ] | Valid D 2.264248e-001 [↓ ] | Stag: 50 Ovfit: 0\n[12/11/2015 22:43:21] 2/2 | Batch 531/590 | D 2.169427e-001 [↑ ] | Valid D 2.203927e-001 [↓ ] | Stag: 60 Ovfit: 0\n[12/11/2015 22:43:22] 2/2 | Batch 541/590 | D 2.233351e-001 [↑ ] | Valid D 2.353653e-001 [↑ ] | Stag: 70 Ovfit: 0\n[12/11/2015 22:43:22] 2/2 | Batch 551/590 | D 3.425132e-001 [↑ ] | Valid D 2.559682e-001 [↑ ] | Stag: 80 Ovfit: 0\n[12/11/2015 22:43:23] 2/2 | Batch 561/590 | D 2.768238e-001 [↓ ] | Valid D 2.412431e-001 [↓ ] | Stag: 90 Ovfit: 0\n[12/11/2015 22:43:24] 2/2 | Batch 571/590 | D 2.550858e-001 [↓ ] | Valid D 2.726600e-001 [↑ ] | Stag:100 Ovfit: 0\n[12/11/2015 22:43:24] 2/2 | Batch 581/590 | D 2.308137e-001 [↓ ] | Valid D 2.466903e-001 [↓ ] | Stag:110 Ovfit: 0\n[12/11/2015 22:43:25] Duration : 00:01:17.5011734\n[12/11/2015 22:43:25] Loss initial : D 2.383214e+000\n[12/11/2015 22:43:25] Loss final : D 1.087980e-001 (Best)\n[12/11/2015 22:43:25] Loss change : D -2.274415e+000 (-95.43 %)\n[12/11/2015 22:43:25] Loss chg. / s : D -2.934685e-002\n[12/11/2015 22:43:25] Epochs / s : 0.02580606089\n[12/11/2015 22:43:25] Epochs / min : 1.548363654\n[12/11/2015 22:43:25] --- Training finished\n\n\n
![\"Chart\"](\"img/Feedforwardnets-3.png\")
\n ![\"Chart\"](\"img/Feedforwardnets-2.png\")
\n \nval pred : int [] = [|5; 1; 9; 2; 6; 0; 0; 5; 7; 6|]\nval real : int [] = [|5; 1; 9; 2; 6; 0; 0; 5; 7; 6|]\n\n\nLet's compute the classification error for the whole MNIST test set of 10,000 examples.\n*)\n\ncc.ClassificationError(MNISTtest)\n\n(**\n
\nval it : float32 = 0.0502999984f\n\n\nThe classification error is around 5%. This can be lowered some more by training the model for more than 2 epochs as we did.\n\nClassifying a single digit:\n*)\n\nlet cls = cc.Classify(MNISTtest.X.[*,0]);;\nMNISTtest.X.[*,0] |> DV.visualizeAsDM 28 |> printfn \"%s\"\n\n(**\n [lang=cs]\n val cls : int = 5\n\n DM : 28 x 28\n \n \n \n \n \n · \n ▴●██♦- \n ▴♦██■▴- \n ♦█■■███▪· \n ■████■- \n ♦███▪ \n ♦██♦ \n ██● \n ■█▪ \n ██· -▴■●- \n ▴██████■███- \n ♦██♦▪ ▪█■- \n ▪· ▴█● \n -██ \n ♦█● \n ■█■ \n -●██■· \n -▴▪■███▪ \n ███████●- \n \n \n \n\nClassifying many digits at the same time:\n*)\n\nlet clss = cc.Classify(MNISTtest.X.[*,0..4]);;\nMNISTtest.[0..4].VisualizeXColsAsImageGrid(28) |> printfn \"%s\"\n\n(**\n\n [lang=cs]\n val clss : int [] = [|5; 1; 9; 2; 6|]\n\n Hype.Dataset\n X: 784 x 5\n Y: 1 x 5\n X's columns reshaped to (28 x 28), presented in a (2 x 3) grid:\n DM : 56 x 84\n \n \n \n \n ██♦ \n · ██ \n ▴●██♦- ██▴ -♦█▪ \n ▴♦██■▴- ♦██ ●█████● \n ♦█■■███▪· ██♦ ■███♦♦██ \n ■████■- ███ ■██♦ ■█▴ \n ♦███▪ ▴███ ·██♦ ●██ \n ♦██♦ ███ ▪██ ■█■ \n ██● ▴██▴ ·██· ·♦██▴ \n ■█▪ ███ ███♦♦████▴ \n ██· -▴■●- ███♦ ▴████████· \n ▴██████■███- ███ ▴ ·-●- ■██ \n ♦██♦▪ ▪█■- ♦██▴ ██■ \n ▪· ▴█● ▴██♦ -██▴ \n -██ ███▴ -██· \n ♦█● ♦██▴ ■██· \n ■█■ ███ ███ \n -●██■· ♦██▴ ▴██● \n -▴▪■███▪ ██♦ ███ \n ███████●- ♦█ -██■ \n -██♦ \n -██· \n \n \n \n \n ▴●█♦ \n ●██ -████▴ \n ▪████● ▴████ \n ▴██████▴ ▴███■ \n ■██▪▴██▴ ███▪ \n ▴██● ▴█■ ■██▴ \n ·███ ██- ·♦██▴ \n ♦██● ▪█▪ -███▴ \n ███ ██ ███♦ \n ███ █♦ ███▪ \n █♦· █♦ ●██■ ▴▴▴ \n · ██ ███ -██-■█████▪ \n - ██ ██■ ●███████████- \n ·██■♦- ██ ▴██▴ ███●- ▪██▴ \n ♦█████■███ ▪██ ·██- ·██▪ \n ■█████████ ███▪·██▴ ♦██ \n ♦█████████♦▪ ▪██████▴ ·♦██· \n -███████████■●●●· ▪███████████████▴ \n ■██████■■■█████▴ -▪██████████♦- \n ·████■ ▴████■ ·▴▴▴▴▴▴· \n -■█- ■■▴ \n \n \n \n \n \n\n\n\n\nNested optimization of training hyperparameters\n-----------------------------------------------\n\nAs we've seen in [optimization](optimization.html), nested AD allows us to apply gradient-based optimization to functions that also internally perform optimization.\n\nThis gives us the possibility of optimizing the hyperparameters of training. We can, for example, compute the gradient of the final loss of a training procedure with respect to the continuous hyperparameters of the training such as learning rates, momentum parameters, regularization coefficients, or initialization conditions. \n\nAs an example, let's train a neural network with a learning rate schedule of 50 elements, and optimize this schedule vector with another level of optimization on top of the training.\n*)\n\nlet train lrschedule =\n Rnd.Seed(123)\n n.Init()\n\n let p = {Params.Default with\n LearningRate = Schedule lrschedule\n Loss = CrossEntropyOnSoftmax\n ValidationInterval = 1\n Silent = true\n ReturnBest = false\n Batch = Full}\n let loss, _ = n.Train(MNISTvalid.[..20], p)\n loss\n\nlet hypertrain epochs =\n let p = {Params.Default with \n Epochs = epochs\n LearningRate = RMSProp(D 0.01f, D 0.9f)\n ValidationInterval = 1}\n let lr, _, _, _ = Optimize.Minimize(train, DV.create 50 (D 0.1f), p)\n lr\n\nlet lr = hypertrain 50\n\n(*** hide ***)\nopen RProvider\nopen RProvider.graphics\nopen RProvider.grDevices\n\nlet lrlr = lr |> DV.toArray |> Array.map (float32>>float)\n\nnamedParams[\n \"x\", box lrlr\n \"pch\", box 19\n \"col\", box \"darkblue\"\n \"type\", box \"o\"\n \"xlab\", box \"Iteration\"\n \"ylab\", box \"Learning rate\"\n \"width\", box 700\n \"height\", box 500\n ]\n|> R.plot|> ignore\n\n(**\n
![\"Chart\"](\"img/Feedforwardnets-4.png\")
\n \n[25/12/2015 23:53:10] --- Minimization started\n[25/12/2015 23:53:10] Parameters : 1\n[25/12/2015 23:53:10] Iterations : 10\n[25/12/2015 23:53:10] Valid. interval: 1\n[25/12/2015 23:53:10] Method : Exact Newton\n[25/12/2015 23:53:10] Learning rate : Constant a = D 1.0f\n[25/12/2015 23:53:10] Momentum : None\n[25/12/2015 23:53:10] Gradient clip. : None\n[25/12/2015 23:53:10] Early stopping : None\n[25/12/2015 23:53:10] Improv. thresh.: D 0.995000005f\n[25/12/2015 23:53:10] Return best : true\n[25/12/2015 23:53:10] 1/10 | D 2.535113e+000 [- ]\n[25/12/2015 23:53:10] 2/10 | D 7.528733e-002 [↓▼]\n[25/12/2015 23:53:10] 3/10 | D 1.592970e-002 [↓▼]\n[25/12/2015 23:53:10] 4/10 | D 4.178338e-003 [↓▼]\n[25/12/2015 23:53:10] 5/10 | D 1.382800e-008 [↓▼]\n[25/12/2015 23:53:11] 6/10 | D 3.274181e-011 [↓▼]\n[25/12/2015 23:53:11] 7/10 | D 1.151079e-012 [↓▼]\n[25/12/2015 23:53:11] 8/10 | D 1.151079e-012 [- ]\n[25/12/2015 23:53:11] 9/10 | D 1.151079e-012 [- ]\n[25/12/2015 23:53:11] 10/10 | D 3.274181e-011 [↑ ]\n[25/12/2015 23:53:11] Duration : 00:00:00.9201285\n[25/12/2015 23:53:11] Value initial : D 2.535113e+000\n[25/12/2015 23:53:11] Value final : D 1.151079e-012 (Best)\n[25/12/2015 23:53:11] Value change : D -2.535113e+000 (-100.00 %)\n[25/12/2015 23:53:11] Value chg. / s : D -2.755173e+000\n[25/12/2015 23:53:11] Iter. / s : 10.86804723\n[25/12/2015 23:53:11] Iter. / min : 652.0828341\n[25/12/2015 23:53:11] --- Minimization finished\n\nval whist : DV [] =\n [|DV [|0.0f|]; DV [|0.20767726f|]; DV [|0.17457059f|]; DV [|0.190040559f|];\n DV [|0.182180524f|]; DV [|0.182166189f|]; DV [|0.182166889f|];\n DV [|0.182166755f|]; DV [|0.182166621f|]; DV [|0.182166487f|]|]\nval w : DV = DV [|0.182166889f|]\nval lhist : D [] =\n [|D 2.5351131f; D 2.5351131f; D 0.0752873272f; D 0.0159297027f;\n D 0.00417833822f; D 1.38279992e-08f; D 3.27418093e-11f; D 1.15107923e-12f;\n D 1.15107923e-12f; D 1.15107923e-12f|]\nval l : D = D 1.15107923e-12f\n\n*)\n\n(*** hide ***)\nopen RProvider\nopen RProvider.graphics\nopen RProvider.grDevices\n\nR.plot_new (namedParams [ ])\n\nlet t = trajectory (whist.[1].[0])\nlet tx, ty = t |> Seq.toArray |> Array.map (fun v -> v.[0] |> float32 |> float, v.[1] |> float32 |> float) |> Array.unzip\n\nnamedParams[\n \"x\", box tx\n \"y\", box ty\n \"pch\", box 1\n \"xlab\", box \"\"\n \"ylab\", box \"\"\n \"col\", box \"darkblue\"\n \"type\", box \"l\"\n \"lty\", box 4\n \"width\", box 700\n \"height\", box 500\n ]\n|> R.lines |> ignore\n\n\n(**\n
![\"Chart\"](\"img/Optimization-3.png\")
\n \n[12/11/2015 01:22:59] --- Minimization started\n[12/11/2015 01:22:59] Parameters : 2\n[12/11/2015 01:22:59] Iterations : 3000\n[12/11/2015 01:22:59] Valid. interval: 10\n[12/11/2015 01:22:59] Method : Gradient descent\n[12/11/2015 01:22:59] Learning rate : RMSProp a0 = D 0.00999999978f, k = D 0.899999976f\n[12/11/2015 01:22:59] Momentum : None\n[12/11/2015 01:22:59] Gradient clip. : None\n[12/11/2015 01:22:59] Early stopping : None\n[12/11/2015 01:22:59] Improv. thresh.: D 0.995000005f\n[12/11/2015 01:22:59] Return best : true\n[12/11/2015 01:22:59] 1/3000 | D 4.125000e+001 [- ]\n[12/11/2015 01:22:59] 11/3000 | D 2.655878e+001 [↓▼]\n[12/11/2015 01:22:59] 21/3000 | D 2.154373e+001 [↓▼]\n[12/11/2015 01:22:59] 31/3000 | D 1.841705e+001 [↓▼]\n[12/11/2015 01:22:59] 41/3000 | D 1.624916e+001 [↓▼]\n[12/11/2015 01:22:59] 51/3000 | D 1.465973e+001 [↓▼]\n[12/11/2015 01:22:59] 61/3000 | D 1.334291e+001 [↓▼]\n...\n[12/11/2015 01:22:59] 2921/3000 | D 9.084024e-004 [- ]\n[12/11/2015 01:22:59] 2931/3000 | D 9.084024e-004 [- ]\n[12/11/2015 01:22:59] 2941/3000 | D 9.084024e-004 [- ]\n[12/11/2015 01:22:59] 2951/3000 | D 9.084024e-004 [- ]\n[12/11/2015 01:22:59] 2961/3000 | D 9.084024e-004 [- ]\n[12/11/2015 01:22:59] 2971/3000 | D 9.084024e-004 [- ]\n[12/11/2015 01:22:59] 2981/3000 | D 9.084024e-004 [- ]\n[12/11/2015 01:22:59] 2991/3000 | D 9.084024e-004 [- ]\n[12/11/2015 01:22:59] Duration : 00:00:00.3142646\n[12/11/2015 01:22:59] Value initial : D 4.125000e+001\n[12/11/2015 01:22:59] Value final : D 8.948371e-004 (Best)\n[12/11/2015 01:22:59] Value change : D -4.124910e+001 (-100.00 %)\n[12/11/2015 01:22:59] Value chg. / s : D -1.312560e+002\n[12/11/2015 01:22:59] Iter. / s : 9546.09587\n[12/11/2015 01:22:59] Iter. / min : 572765.7522\n[12/11/2015 01:22:59] --- Minimization finished\n\nval wopt : DV = DV [|2.99909306f; 0.50039643f|]\n\n*)\n\n(*** hide ***)\nopen RProvider\nopen RProvider.graphics\nopen RProvider.grDevices\n\nR.plot_new (namedParams [ ])\n\nlet ll = lhist |> Array.map (float32>>float)\n\nnamedParams[\n \"x\", box ll\n \"pch\", box 19\n \"col\", box \"darkblue\"\n \"type\", box \"o\"\n \"xlab\", box \"Iteration\"\n \"ylab\", box \"Function value\"\n \"width\", box 700\n \"height\", box 500\n ]\n|> R.plot|> ignore\n\n(**\n
![\"Chart\"](\"img/Optimization-1.png\")
\n ![\"Chart\"](\"img/Optimization-2.png\")
\n \n[|\",\"; \".\"; \":\"; \"?\"; \"Alba\"; \"Can\"; \"Heaven\"; \"I\"; \"Italy\"; \"Juno’s\"; \"Latin\";\n \"Latium\"; \"Lavinian\"; \"Longa\"; \"Muse\"; \"Queen\"; \"Rome\"; \"Troy\"; \"a\"; \"about\";\n \"also\"; \"and\"; \"anger\"; \"arms\"; \"be\"; \"brought\"; \"by\"; \"came\"; \"cause\"; \"city\";\n \"coast\"; \"cruel\"; \"dangers\"; \"divinity\"; \"drive\"; \"endlessly\"; \"endure\";\n \"exiled\"; \"face\"; \"fate\"; \"first\"; \"for\"; \"founded\"; \"from\"; \"gods\"; \"grieved\";\n \"he\"; \"her\"; \"his\"; \"how\"; \"hurled\"; \"in\"; \"land\"; \"long\"; \"lords\"; \"man\";\n \"many\"; \"me\"; \"minds\"; \"noble\"; \"noted\"; \"of\"; \"offended\"; \"people\";\n \"remorseless\"; \"sea\"; \"she\"; \"shores\"; \"sing\"; \"so\"; \"such\"; \"suffering\";\n \"tell\"; \"that\"; \"the\"; \"there\"; \"to\"; \"trials\"; \"until\"; \"virtue\"; \"walls\";\n \"war\"; \"was\"; \"who\"; \"will\"; \"–\"|]\n \n 86\n\nThere are 86 tokens in this language instance.\n\nNow let's transform the full text to a dataset, using the **Language** instance holding these tokens. The text will be encoded in a matrix where each column is a representation of each word as a _one-hot_ vector.\n*)\n\nlet text' = lang.EncodeOneHot(text)\ntext'.Visualize() |> printfn \"%s\"\n\n(**\n
\nDM : 86 x 145\n\n\nOut of these 145 words, we will construct a dataset where the inputs are the first 144 words and the target outputs are the 144 words starting with a one word shift. This means that, for each word, we want the output (the prediction) to be the following word in our text passage.\n*)\n\nlet data = Dataset(text'.[*, 0..(text'.Cols - 2)],\n text'.[*, 1..(text'.Cols - 1)])\n\n(**\n
\nval data : Dataset = Hype.Dataset\n X: 86 x 144\n Y: 86 x 144\n\n\nRNNs, and especially the LSTM variety that we will use, can make predictions that take long-term dependencies and contextual information into account. When the language model is trained with a large enough text corpus and the network has enough capacity, state-of-the-art RNN language models are able to learn complex grammatical relations.\n\nFor our quick demonstration, we use a linear word embedding layer of 20 units, an LSTM of 100 units and a final linear layer of 86 units (the size of our vocabulary) followed by **softmax** activation.\n*)\n\nlet dim = lang.Length // Vocabulary size, here 86\n\nlet n = FeedForward()\nn.Add(Linear(dim, 20))\nn.Add(LSTM(20, 100))\nn.Add(Linear(100, dim))\nn.Add(DM.mapCols softmax)\n\n(**\nYou can also easily stack multiple RNNs on top of each other.\n*)\n\nlet n = FeedForward()\nn.Add(Linear(dim, 20))\nn.Add(LSTM(20, 100))\nn.Add(LSTM(100, 100))\nn.Add(Linear(100, dim))\nn.Add(DM.mapCols softmax)\n\n(**\nWe will observe the the performance of our RNN during training by sampling random sentences from the language model. \n\nRemember that the final output of the network, through the softmax activation, is a vector of word probabilities. When we are sampling, we start with a word, supply this to the network, and use the resulting probabilities at the output to sample from the vocabulary where words with higher probability are more likely to be selected. We then continue by giving the network the last sampled word and repeating this until we hit an \"end of sentence\" token (we use \".\" here) or reach a limit of maximum sentence length.\n\nThis is how we would sample a sentence starting with a specific word.\n*)\n\nn.Reset()\nfor i = 0 to 5 do\n lang.Sample(n.Run, \"I\", [|\".\"|], 30) // Use \".\" as the stop token, limit maximum sentence length to 30.\n |> printfn \"%s\"\n\n(**\n\nBecause the model is not trained, we get sequences of random words from the vocabulary.\n\n
\nI be: she dangers Latium endlessly gods remorseless divinity tell and his offended lords trials? about war trials and anger shores so anger Alba a Alba sing her\nI? came exiled – suffering shores anger came Latium people sing sing remorseless who brought war walls endlessly anger me founded his.\nI – will long of in offended cruel until Queen Italy who anger lords Queen in Longa Muse who people about suffering Italy also grieved cruel hurled who me about\nI endlessly city first by face, a Heaven me hurled sea such long noted she noted many sea city anger I noted remorseless cause Queen to remorseless Italy coast\nI sea noted noble me minds long sing cause people in walls Italy by Longa first, for grieved sea many walls Troy came was endlessly of in Latium Latium\nI and Latin of many suffering Alba Latium war.\n\n\nWe set a training cycle where we run one epoch of training followed by sampling one sentence starting with the word \"I\". In each epoch, we run through the whole training dataset. With a larger training corpus, we could also run the training with minibatches by stating this in the parameter set (commented out below).\n\nLike the sample sentences above, at the beginning of training, we see mostly random orderings of words. As the training progresses, the cross-entropy loss for our dataset is decreasing and the sentences start exhibiting meaningful word patterns.\n*)\n\nfor i = 0 to 1000 do\n let par = {Params.Default with\n //Batch = Minibatch 10\n LearningRate = LearningRate.RMSProp(D 0.01f, D 0.9f)\n Loss = CrossEntropyOnSoftmax\n Epochs = 1\n Silent = true // Suppress the regular printing of training progress\n ReturnBest = false} \n let loss, _ = Layer.Train(n, data, par)\n printfn \"Epoch: %*i | Loss: %O | Sample: %s\" 3 i loss (lang.Sample(n.Run, \"I\", [|\".\"|], 30))\n\n(**\n\nHere is a selection of sentences demonstrating the progress of training.\n\n
\nEpoch: 0 | Loss: D 4.478101e+000 | Sample: I Queen drive she Alba endlessly Queen the by how tell his from grieved war her there drive people – lords coast he.\nEpoch: 10 | Loss: D 4.102071e+000 | Sample: I people to,, Rome how the he of – sing fate, Muse, by,, Muse the of man Queen Latin and in her cause:\nEpoch: 30 | Loss: D 3.438288e+000 | Sample: I walls long to first dangers she her, to founded to virtue sea first Can dangers a founded about Can Queen lords from sea by remorseless founded endlessly Latium\nEpoch: 40 | Loss: D 2.007577e+000 | Sample: I Alba gods Alba Rome, the walls Alba Muse Rome anger me the the of the gods to who man me first founded offended endlessly until also grieved long\nEpoch: 50 | Loss: D 9.753818e-001 | Sample: I sing people cruel: me the of Rome.\nEpoch: 60 | Loss: D 3.944587e-001 | Sample: I sing sing Troy to so hurled endlessly by land sea, by to – hurled about by the of arms, by Juno’s such anger long also in her\nEpoch: 70 | Loss: D 2.131431e-001 | Sample: I sing of and the of Longa, by Juno’s anger was in her of Heaven, to a city brought his gods to a gods to Lavinian hurled to\nEpoch: 80 | Loss: D 1.895453e-001 | Sample: I sing, by will the of Rome.\nEpoch: 90 | Loss: D 1.799535e-001 | Sample: I sing? there Muse the of the of the of arms by the: how she offended in the of? a, he shores hurled by land to\nEpoch: 100 | Loss: D 1.733837e-001 | Sample: I sing arms the of Alba gods who, by Juno’s Rome such anger the of the of arms and, by, by from the coast Rome.\nEpoch: 110 | Loss: D 1.682917e-001 | Sample: I sing Troy by, by from the of arms and, by, by from came, by Juno’s anger long in the of the of arms cruel Muse\nEpoch: 120 | Loss: D 1.639529e-001 | Sample: I sing arms the of Rome.\nEpoch: 130 | Loss: D 1.600647e-001 | Sample: I sing arms and, by Juno’s remorseless there and the of the of arms and, by Alba coast Troy to a – his gods by of the of\nEpoch: 140 | Loss: D 1.564835e-001 | Sample: I sing arms by the of Rome.\nEpoch: 150 | Loss: D 1.531392e-001 | Sample: I sing arms cruel, exiled by coast, he a city in the of the of arms.\nEpoch: 160 | Loss: D 1.499920e-001 | Sample: I sing arms cruel man, by the trials arms to shores hurled endlessly by the of gods Italy, me the of Rome.\nEpoch: 200 | Loss: D 1.390327e-001 | Sample: I sing arms and, by Juno’s such of the of the of arms Italy, by from the sing arms walls of the of Rome.\nEpoch: 230 | Loss: D 1.322940e-001 | Sample: I sing arms the man he, tell from the of arms Italy, by fate, by the of Troy Italy, by fate first from the of the\nEpoch: 260 | Loss: D 1.264137e-001 | Sample: I sing brought Muse Muse the of Heaven, by shores remorseless there he in the of arms cruel, by fate, he from the gods to Italy,\nEpoch: 420 | Loss: D 1.131158e-001 | Sample: I sing of arms the of Heaven, by Juno’s remorseless hurled such in the of arms.\nEpoch: 680 | Loss: D 9.938217e-002 | Sample: I of arms the man he, exiled fate, he virtue, to a? Can be such in the of the of of the of arms.\nEpoch: 923 | Loss: D 9.283429e-002 | Sample: I sing of arms and the man he, by fate came from the of to Italy, by the, by Juno’s anger of Rome.\n\n*)\n" }, { "path": "docs/input/Regression.fsx", "content": "(*** hide ***)\n#r \"../../src/Hype/bin/Release/netstandard2.0/DiffSharp.dll\"\n#r \"../../src/Hype/bin/Release/netstandard2.0/Hype.dll\"\n#I \"../../packages/R.NET.Community/lib/net40/\"\n#I \"../../packages/R.NET.Community.FSharp/lib/net40/\"\n#I \"../../packages/RProvider\"\n#load \"RProvider.fsx\"\nfsi.ShowDeclarationValues <- true\n\n(**\nRegression\n==========\n\nIn this example we implement a logistic regression based binary classifier and train it to distinguish between the [MNIST](http://yann.lecun.com/exdb/mnist/) digits of 0 and 1.\n\n### Loading the data\n\nFirst, let's start by loading the MNIST training and testing data and arranging these into training, validation, and testing sets.\n*)\n\nopen Hype\nopen Hype.Neural\nopen DiffSharp.AD.Float32\nopen DiffSharp.Util\n\nlet MNIST = Dataset(Util.LoadMNISTPixels(\"C:/datasets/MNIST/train-images.idx3-ubyte\", 60000),\n Util.LoadMNISTLabels(\"C:/datasets/MNIST/train-labels.idx1-ubyte\", 60000) |> toDV |> DM.ofDV 1).NormalizeX()\n\n\n\nlet MNISTtrain = MNIST.[..58999]\nlet MNISTvalid = MNIST.[59000..]\n\nlet MNISTtest = Dataset(Util.LoadMNISTPixels(\"C:/datasets/MNIST/t10k-images.idx3-ubyte\", 10000),\n Util.LoadMNISTLabels(\"C:/datasets/MNIST/t10k-labels.idx1-ubyte\", 10000) |> toDV |> DM.ofDV 1).NormalizeX()\n\n(**\nWe shuffle the columns of the datasets and filter them to only keep the digits of 0 and 1.\n*)\n\nlet MNISTtrain01 = MNISTtrain.Shuffle().Filter(fun (x, y) -> y.[0] <= D 1.f)\nlet MNISTvalid01 = MNISTvalid.Shuffle().Filter(fun (x, y) -> y.[0] <= D 1.f)\nlet MNISTtest01 = MNISTtest.Shuffle().Filter(fun (x, y) -> y.[0] <= D 1.f)\n\n(**\n
\nval MNISTtrain01 : Dataset = Hype.Dataset\n X: 784 x 12465\n Y: 1 x 12465\nval MNISTvalid01 : Dataset = Hype.Dataset\n X: 784 x 200\n Y: 1 x 200\nval MNISTtest01 : Dataset = Hype.Dataset\n X: 784 x 2115\n Y: 1 x 2115\n\n\nWe can visualize individual digits from the dataset.\n*)\n\nMNISTtrain.X.[*,9] |> DV.visualizeAsDM 28 |> printfn \"%s\"\nMNISTtrain.Y.[*,9]\n\n(**\n [lang=cs]\n DM : 28 x 28\n \n \n \n \n ♦♦ \n ▪█▪ \n ▴██· \n ♦█♦ \n ● ·█■ \n ■█ ■█· \n ♦█ ██· \n ▴█■ ●█♦ \n ■█ ▪█■ \n ■█▪ ▴██- \n -███♦▴ ♦█▪ \n ·███■██♦■█■ \n ·██■ ♦█████■ \n ♦■- ♦████▪ \n - ██· \n ▪█■ \n ▴█■ \n ■█▴ \n ■█▪ \n ▴█· \n \n \n \n \n\n val it : DV = DV [|4.0f|]\n\nWe can also visualize a series of digits in grid layout.\n*)\n\nMNISTtrain.[..5].VisualizeXColsAsImageGrid(28) |> printfn \"%s\"\n\n(**\n [lang=cs]\n Hype.Dataset\n X: 784 x 6\n Y: 1 x 6\n X's columns reshaped to (28 x 28), presented in a (2 x 3) grid:\n DM : 56 x 84\n \n \n \n \n ▪█▪ \n ▴▴● ●██▴ █████ ■ \n -▪●█████■●██♦ ■███■█ · ▴● \n ██████████-·· ■███♦·██▴ ▴● ▪♦ \n ■█████♦●██ ●██████-♦█● ■● █▪ \n ·▪-██♦ ▪ ███♦-█■ ·█● ■● ●█▴ \n ▪█· ███● ·▴ ██ █● ♦█ \n ▴█♦ ●█■♦· ██● ▴█● ■█ \n ♦█· ●██· ██♦ ▪█▴ ●█■ \n █■▪- ██ ██♦ ▪█ ·●██· \n ·███▴ ♦█♦ ██♦ ▪█· ▴▪▪███●██ \n ●██▪ ·██- ██▪ ▪██♦♦♦████♦▪· ■█ \n -██♦ ·█■ ▴█● ▴●●●●●- -█■ \n ███ ·█■ ▴█■· ●█▴ \n ▴●██♦ ·█▪ ●█● ●█ \n ▪■████● ·█■ -██▪ ●█ \n -■████♦· ·██▪ ·●■█■● ●█- \n ■████♦· ·███■■███♦▴ ●█- \n ●■████♦· ♦█████■▪ ●█▪ \n ●■█████▴ ▴███▪ ●█▪ \n ▴███■▴▴ -█▪ \n \n \n \n \n \n \n \n \n ▴██ -▴ \n -███ ▪♦███▪ \n ▴███ ▪♦██-·▪ ▪███■■█■ \n ██■ ·■██♦♦███● ▪████■ ██ \n ■██- ██♦ ●██▴ -████■ ██ \n ▪██♦ -██● ·██■ ●██■● ·██ \n ███ ▴██▪ ■██· ▴ -██ \n ♦██▴ ▴██● ·██▴ ▪██ \n -██● ■█● ♦██● -▴▴▴♦█♦ \n ·██♦ ██ ▴♦████· ●█████■ \n ███▪ ■█████■■█■ ■██■●███■▴ \n ▪███ ██■▴ ♦█▪ ·██▴ ♦████· \n ■██● ██- ▴██● ♦██▴●██● \n ███♦ ·██ ▴██- ♦█■ ·●███■ \n ███· ██ -██· ·●██▴ ·●● \n ▪███ ·██ ■██♦■███▴ \n ■██▪ -██ ♦████●▴ \n ██■ ██ -▪▴ \n ██■ ■█ \n ♦█■ -█♦ \n ●█● \n ▪█ \n \n*)\n\nMNISTtrain01.[..5].VisualizeXColsAsImageGrid(28) |> printfn \"%s\"\n\n(**\n [lang=cs]\n Hype.Dataset\n X: 784 x 6\n Y: 1 x 6\n X's columns reshaped to (28 x 28), presented in a (2 x 3) grid:\n DM : 56 x 84\n \n \n \n \n ▴●███- ·♦██ \n ·▪■█████■ ████● -▪██████ \n ▪████████- ●████■ ▪██████████ \n ●████▪ ●███▴ ■████● ■███■▪▪▴-███● \n ████● ▴███· ██████ -███■ ██████ \n ███▪ ██▴ -██████ ■██▴ ████▴♦- \n ■██· ●█● ▪█████■ ■██ ■█▴█■ \n ●██· ▴█● ██████ ▪██● - ■█- \n -██▪ ▴█● ██████ ██■ ▪█▴ \n ██■ ▴█● ■█████▴ ▴██- ■█▴ \n ·██ ▴█● ██████- ▴██ ██▴ \n ■█■ ▴█● █████▴ ▴█■ ██ \n ■█▪ ▴█● ♦████- ▴█▪ ▴█▪ \n ██- ●█● ♦█████ ▴█▪ ●█· \n ██▴ ██· █████♦ -██ -█▪ \n ███ ●██· ●█████· ██■ █- \n -███● ▴██● ·█████ ███ ■■ \n ■█████♦●●■███■ ♦█████ ██- -■█· \n ▴■█████████♦ █████♦ ●██● -■██▪ \n ·▪▪●█♦▪▴ ▴████· ●████████■- \n ▪█████▪- \n \n \n \n \n \n \n \n ■▴ ····· \n ▴■ ▪■█████· \n ♦♦♦♦♦-· ■█ ▴█████████- \n ▪████████■▪ ██ ███████████· \n ■███████████■ ▪█■ ▴■████♦♦ ▴████ \n █████████ ♦███ ██- ▴████♦▴ ■███· \n ▪███● ● ·██■ █● ▴████■ ♦███- \n ██■ -██· ▪█▴ ████▪ ████ \n ▪██ ██♦ ██ ███- ████ \n ●█♦ ▪██ ▪██ ·███· ████ \n ██▴ ▴██ ●██ ·███♦ ▪████ \n ██ ♦█▪ ██● ·███ ▪████· \n ██- ██ ██· ●███ ·●█████· \n ♦█● ■██ ·██- ████· ■██████▴ \n ▪█■ ■██- █♦ ████■▴▴▴■████████▪ \n -██● ▴███♦ -█♦ ■██████████████♦ \n ♦██♦-▪ ▪♦██■- ■█♦ ·████████████▪▴ \n ♦███████████● ■█- -█████████▴ \n ▴■█■■■■■■· █■ ▪ ▪▪ \n ♦▴ \n \n \n \n \n### Defining the model\n\nLet's now create our linear regression model. We implement this using the **Hype.Neural** module, as a linear layer with $28 \\times 28 = 784$ inputs and one output. The output of the layer is passed through the sigmoid function.\n\n*)\n\nlet n = Neural.FeedForward()\nn.Add(Linear(28 * 28, 1))\nn.Add(sigmoid)\n\n(**\n\nWe can visualize the initial state of the linear model weights before the training. For information of about weight initialization parameters, please see the [neural networks example](feedforwardnets.html).\n\n*)\n\nlet l = (n.[0] :?> Linear)\nl.VisualizeWRowsAsImageGrid(28) |> printfn \"%s\"\n\n(** \n\n [lang=cs]\n Hype.Neural.Linear\n 784 -> 1\n Learnable parameters: 785\n Init: Standard\n W's rows reshaped to (28 x 28), presented in a (1 x 1) grid:\n DM : 28 x 28\n ▴▪●●-█▴♦♦● ·▴█● ● ▴· ●●●●▪·\n ■ █- ▴●●▪ ■♦· ■▪■▪ █ ♦■●■\n ♦■ █♦●▪●♦ ♦■ ♦ ■ ▪- ■\n ■▪ ■♦■♦ █ ▪● ♦▪▴··■█ -▴●▪▪●\n ██··▴●●█▪♦■ -·█■ ▪- ··▪· ██\n - ▪ ♦ ▪● ▪■█♦- ▴▪ ▴· ▪·●\n - ●●▴▴ ▪■ ▴█ ▪▴·▴▴·♦■■♦·■■\n ♦▴ ▪■ ▪▪▴■·■--▪♦- ·♦▪■ ♦·●\n ·▴·♦▪♦●▪··▴·▪ ● ▪ █ ▴▪·♦▪ \n ■ ▴ ♦█▴ - ♦●■ █▪■●▪█■▴●--█\n ♦■ ●■▴♦ ●· █· ▴· -█-▪●■■-■\n █-·▪▴-▴█ ♦ █●·♦█▪▪●●■ - ·\n - █ ■♦·●▪▴♦ -▴ - ■♦· ♦ -\n ■█ ▪- ▪■●♦█▴-█▪■ ■♦▪█■▪■ -\n ●♦█▴♦♦ ♦ ▴▪▴▴♦-▴♦♦█ ▴ ▪·● \n ·█▪■■█ ●· ●· -●■●·· ▴ --▴\n ·♦█▴ ♦♦■ ▴▪●▪- · -♦●♦ ■ · ■\n ■■▪---♦■·●▴▪-▪▴· ▪●● ·♦■ ▪♦▴\n ▴ -♦●■█·█ ● ♦▪●■- ·■♦-▪▴■▴\n ●-■● ···●█▴▪ -█·▪ ♦▴ ● ●\n ·█ █▴ ·♦---■▴·█■■▴ ▴■ - █\n - ▪ ●█·▴♦▪ ■ ▪■ ■··· ▴\n ■ ♦♦- █▪♦-- ▴ ▴ ··█▴● ■♦ \n ■·■■▪▴-·█♦●■ ▴ ♦ ♦▴■♦ ■ ●♦▪\n ·█▪- ■●▴▪▴▪ ▪ ▴▪ · ▪▴▴··♦\n ▪█♦■ ·♦ ■▪ ♦ ▴·●█▪· ·▪▴ \n · ■♦▪■ ▪■● ♦ ··· ·▪█■· ▪■●\n ●▴▪ ·■● -█●█·▪■▴ ▴▴♦ ■ ■ ▴\n\n b:\n DV : 1\n \n\n### Training\n\nLet's train the model for 10 epochs (full passes through the training data), with a minibatch size of 100, using the training and validation sets we've defined. The validation set will make sure that we're not overfitting the model.\n*)\n\nlet p = {Params.Default with \n Epochs = 10; \n Batch = Minibatch 100; \n EarlyStopping = EarlyStopping.DefaultEarly}\n\nn.Train(MNISTtrain01, MNISTvalid01, p)\n\n(**\n
\n[12/11/2015 20:21:12] --- Training started\n[12/11/2015 20:21:12] Parameters : 785\n[12/11/2015 20:21:12] Iterations : 1240\n[12/11/2015 20:21:12] Epochs : 10\n[12/11/2015 20:21:12] Batches : Minibatches of 100 (124 per epoch)\n[12/11/2015 20:21:12] Training data : 12465\n[12/11/2015 20:21:12] Validation data: 200\n[12/11/2015 20:21:12] Valid. interval: 10\n[12/11/2015 20:21:12] Method : Gradient descent\n[12/11/2015 20:21:12] Learning rate : RMSProp a0 = D 0.00100000005f, k = D 0.899999976f\n[12/11/2015 20:21:12] Momentum : None\n[12/11/2015 20:21:12] Loss : L2 norm\n[12/11/2015 20:21:12] Regularizer : L2 lambda = D 9.99999975e-05f\n[12/11/2015 20:21:12] Gradient clip. : None\n[12/11/2015 20:21:12] Early stopping : Stagnation thresh. = 750, overfit. thresh. = 10\n[12/11/2015 20:21:12] Improv. thresh.: D 0.995000005f\n[12/11/2015 20:21:12] Return best : true\n[12/11/2015 20:21:12] 1/10 | Batch 1/124 | D 4.748471e-001 [- ] | Valid D 4.866381e-001 [- ] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:12] 1/10 | Batch 11/124 | D 2.772053e-001 [↓▼] | Valid D 3.013612e-001 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:12] 1/10 | Batch 21/124 | D 2.178165e-001 [↓▼] | Valid D 2.304372e-001 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:12] 1/10 | Batch 31/124 | D 2.009703e-001 [↓▼] | Valid D 1.799015e-001 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:12] 1/10 | Batch 41/124 | D 1.352896e-001 [↓▼] | Valid D 1.405802e-001 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:12] 1/10 | Batch 51/124 | D 1.182899e-001 [↓▼] | Valid D 1.108390e-001 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:12] 1/10 | Batch 61/124 | D 1.124191e-001 [↓▼] | Valid D 8.995526e-002 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:12] 1/10 | Batch 71/124 | D 8.975799e-002 [↓▼] | Valid D 7.361954e-002 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:12] 1/10 | Batch 81/124 | D 5.031444e-002 [↓▼] | Valid D 5.941865e-002 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:12] 1/10 | Batch 91/124 | D 5.063754e-002 [↑ ] | Valid D 4.927430e-002 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:12] 1/10 | Batch 101/124 | D 3.842642e-002 [↓▼] | Valid D 4.095582e-002 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:12] 1/10 | Batch 111/124 | D 4.326219e-002 [↑ ] | Valid D 3.452797e-002 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:12] 1/10 | Batch 121/124 | D 2.585407e-002 [↓▼] | Valid D 2.788338e-002 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:12] 2/10 | Batch 1/124 | D 3.069563e-002 [↑ ] | Valid D 2.663207e-002 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:12] 2/10 | Batch 11/124 | D 1.765305e-002 [↓▼] | Valid D 2.332163e-002 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:12] 2/10 | Batch 21/124 | D 2.314118e-002 [↑ ] | Valid D 1.902804e-002 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:12] 2/10 | Batch 31/124 | D 3.177435e-002 [↑ ] | Valid D 1.691620e-002 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:12] 2/10 | Batch 41/124 | D 2.219648e-002 [↓ ] | Valid D 1.455527e-002 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:12] 2/10 | Batch 51/124 | D 1.205402e-002 [↓▼] | Valid D 1.240637e-002 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:12] 2/10 | Batch 61/124 | D 3.891717e-002 [↑ ] | Valid D 1.189688e-002 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:12] 2/10 | Batch 71/124 | D 2.114762e-002 [↓ ] | Valid D 1.083007e-002 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:12] 2/10 | Batch 81/124 | D 5.075417e-003 [↓▼] | Valid D 9.630994e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:12] 2/10 | Batch 91/124 | D 1.343214e-002 [↑ ] | Valid D 8.666289e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:13] 2/10 | Batch 101/124 | D 6.054885e-003 [↓ ] | Valid D 8.039203e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:13] 2/10 | Batch 111/124 | D 1.964125e-002 [↑ ] | Valid D 7.339509e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:13] 2/10 | Batch 121/124 | D 4.401092e-003 [↓▼] | Valid D 6.376633e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:13] 3/10 | Batch 1/124 | D 7.068173e-003 [↑ ] | Valid D 6.426438e-003 [↑ ] | Stag: 10 Ovfit: 0\n[12/11/2015 20:21:13] 3/10 | Batch 11/124 | D 3.763680e-003 [↓▼] | Valid D 6.076077e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:13] 3/10 | Batch 21/124 | D 9.855231e-003 [↑ ] | Valid D 5.091224e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:13] 3/10 | Batch 31/124 | D 1.263964e-002 [↑ ] | Valid D 4.641499e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:13] 3/10 | Batch 41/124 | D 1.205439e-002 [↓ ] | Valid D 4.599225e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:13] 3/10 | Batch 51/124 | D 2.941387e-003 [↓▼] | Valid D 4.381890e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:13] 3/10 | Batch 61/124 | D 2.546543e-002 [↑ ] | Valid D 4.439059e-003 [↑ ] | Stag: 10 Ovfit: 0\n[12/11/2015 20:21:13] 3/10 | Batch 71/124 | D 9.878366e-003 [↓ ] | Valid D 4.358966e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:13] 3/10 | Batch 81/124 | D 1.868963e-003 [↓▼] | Valid D 3.960044e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:13] 3/10 | Batch 91/124 | D 7.171181e-003 [↑ ] | Valid D 3.634899e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:13] 3/10 | Batch 101/124 | D 2.681098e-003 [↓ ] | Valid D 3.636524e-003 [↑ ] | Stag: 10 Ovfit: 0\n[12/11/2015 20:21:13] 3/10 | Batch 111/124 | D 1.502046e-002 [↑ ] | Valid D 3.393996e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:13] 3/10 | Batch 121/124 | D 2.381395e-003 [↓ ] | Valid D 3.178693e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:13] 4/10 | Batch 1/124 | D 3.185510e-003 [↑ ] | Valid D 3.240891e-003 [↑ ] | Stag: 10 Ovfit: 0\n[12/11/2015 20:21:13] 4/10 | Batch 11/124 | D 2.029225e-003 [↓ ] | Valid D 3.163968e-003 [↓ ] | Stag: 20 Ovfit: 0\n[12/11/2015 20:21:13] 4/10 | Batch 21/124 | D 6.450378e-003 [↑ ] | Valid D 2.772849e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:13] 4/10 | Batch 31/124 | D 7.448227e-003 [↑ ] | Valid D 2.572560e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:13] 4/10 | Batch 41/124 | D 9.700718e-003 [↑ ] | Valid D 2.693694e-003 [↑ ] | Stag: 10 Ovfit: 0\n[12/11/2015 20:21:13] 4/10 | Batch 51/124 | D 1.799919e-003 [↓▼] | Valid D 2.737873e-003 [↑ ] | Stag: 20 Ovfit: 1\n[12/11/2015 20:21:13] 4/10 | Batch 61/124 | D 1.919956e-002 [↑ ] | Valid D 2.778393e-003 [↑ ] | Stag: 30 Ovfit: 3\n[12/11/2015 20:21:13] 4/10 | Batch 71/124 | D 5.462923e-003 [↓ ] | Valid D 2.870561e-003 [↑ ] | Stag: 40 Ovfit: 3\n[12/11/2015 20:21:13] 4/10 | Batch 81/124 | D 1.455469e-003 [↓▼] | Valid D 2.632472e-003 [↓ ] | Stag: 50 Ovfit: 4\n[12/11/2015 20:21:14] 4/10 | Batch 91/124 | D 5.270801e-003 [↑ ] | Valid D 2.455564e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:14] 4/10 | Batch 101/124 | D 2.057914e-003 [↓ ] | Valid D 2.511977e-003 [↑ ] | Stag: 10 Ovfit: 0\n[12/11/2015 20:21:14] 4/10 | Batch 111/124 | D 1.314815e-002 [↑ ] | Valid D 2.393763e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:14] 4/10 | Batch 121/124 | D 2.033168e-003 [↓ ] | Valid D 2.358985e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:14] 5/10 | Batch 1/124 | D 2.199435e-003 [↑ ] | Valid D 2.389120e-003 [↑ ] | Stag: 10 Ovfit: 0\n[12/11/2015 20:21:14] 5/10 | Batch 11/124 | D 1.668178e-003 [↓ ] | Valid D 2.356529e-003 [↓ ] | Stag: 20 Ovfit: 0\n[12/11/2015 20:21:14] 5/10 | Batch 21/124 | D 5.649061e-003 [↑ ] | Valid D 2.151499e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:14] 5/10 | Batch 31/124 | D 5.264180e-003 [↓ ] | Valid D 2.038927e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:14] 5/10 | Batch 41/124 | D 8.416546e-003 [↑ ] | Valid D 2.145057e-003 [↑ ] | Stag: 10 Ovfit: 0\n[12/11/2015 20:21:14] 5/10 | Batch 51/124 | D 1.564733e-003 [↓ ] | Valid D 2.208556e-003 [↑ ] | Stag: 20 Ovfit: 0\n[12/11/2015 20:21:14] 5/10 | Batch 61/124 | D 1.581773e-002 [↑ ] | Valid D 2.233998e-003 [↑ ] | Stag: 30 Ovfit: 0\n[12/11/2015 20:21:14] 5/10 | Batch 71/124 | D 3.898179e-003 [↓ ] | Valid D 2.347554e-003 [↑ ] | Stag: 40 Ovfit: 0\n[12/11/2015 20:21:14] 5/10 | Batch 81/124 | D 1.395002e-003 [↓▼] | Valid D 2.182974e-003 [↓ ] | Stag: 50 Ovfit: 1\n[12/11/2015 20:21:14] 5/10 | Batch 91/124 | D 4.450763e-003 [↑ ] | Valid D 2.069927e-003 [↓ ] | Stag: 60 Ovfit: 1\n[12/11/2015 20:21:14] 5/10 | Batch 101/124 | D 1.927794e-003 [↓ ] | Valid D 2.129479e-003 [↑ ] | Stag: 70 Ovfit: 1\n[12/11/2015 20:21:14] 5/10 | Batch 111/124 | D 1.238949e-002 [↑ ] | Valid D 2.059099e-003 [↓ ] | Stag: 80 Ovfit: 1\n[12/11/2015 20:21:14] 5/10 | Batch 121/124 | D 1.969593e-003 [↓ ] | Valid D 2.072177e-003 [↑ ] | Stag: 90 Ovfit: 1\n[12/11/2015 20:21:14] 6/10 | Batch 1/124 | D 1.885590e-003 [↓ ] | Valid D 2.087292e-003 [↑ ] | Stag:100 Ovfit: 1\n[12/11/2015 20:21:14] 6/10 | Batch 11/124 | D 1.577425e-003 [↓ ] | Valid D 2.074389e-003 [↓ ] | Stag:110 Ovfit: 1\n[12/11/2015 20:21:14] 6/10 | Batch 21/124 | D 5.410788e-003 [↑ ] | Valid D 1.943973e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:14] 6/10 | Batch 31/124 | D 4.188792e-003 [↓ ] | Valid D 1.863442e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:14] 6/10 | Batch 41/124 | D 7.516511e-003 [↑ ] | Valid D 1.951990e-003 [↑ ] | Stag: 10 Ovfit: 0\n[12/11/2015 20:21:14] 6/10 | Batch 51/124 | D 1.510475e-003 [↓ ] | Valid D 2.003860e-003 [↑ ] | Stag: 20 Ovfit: 0\n[12/11/2015 20:21:14] 6/10 | Batch 61/124 | D 1.375423e-002 [↑ ] | Valid D 2.020531e-003 [↑ ] | Stag: 30 Ovfit: 0\n[12/11/2015 20:21:14] 6/10 | Batch 71/124 | D 3.260145e-003 [↓ ] | Valid D 2.129138e-003 [↑ ] | Stag: 40 Ovfit: 0\n[12/11/2015 20:21:15] 6/10 | Batch 81/124 | D 1.402565e-003 [↓ ] | Valid D 2.002138e-003 [↓ ] | Stag: 50 Ovfit: 0\n[12/11/2015 20:21:15] 6/10 | Batch 91/124 | D 3.999386e-003 [↑ ] | Valid D 1.920336e-003 [↓ ] | Stag: 60 Ovfit: 0\n[12/11/2015 20:21:15] 6/10 | Batch 101/124 | D 1.929424e-003 [↓ ] | Valid D 1.976652e-003 [↑ ] | Stag: 70 Ovfit: 0\n[12/11/2015 20:21:15] 6/10 | Batch 111/124 | D 1.205915e-002 [↑ ] | Valid D 1.926643e-003 [↓ ] | Stag: 80 Ovfit: 0\n[12/11/2015 20:21:15] 6/10 | Batch 121/124 | D 1.978536e-003 [↓ ] | Valid D 1.951888e-003 [↑ ] | Stag: 90 Ovfit: 0\n[12/11/2015 20:21:15] 7/10 | Batch 1/124 | D 1.769614e-003 [↓ ] | Valid D 1.959661e-003 [↑ ] | Stag:100 Ovfit: 0\n[12/11/2015 20:21:15] 7/10 | Batch 11/124 | D 1.555518e-003 [↓ ] | Valid D 1.955613e-003 [↓ ] | Stag:110 Ovfit: 0\n[12/11/2015 20:21:15] 7/10 | Batch 21/124 | D 5.217655e-003 [↑ ] | Valid D 1.861573e-003 [↓ ] | Stag:120 Ovfit: 0\n[12/11/2015 20:21:15] 7/10 | Batch 31/124 | D 3.625835e-003 [↓ ] | Valid D 1.796666e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:15] 7/10 | Batch 41/124 | D 6.929778e-003 [↑ ] | Valid D 1.872346e-003 [↑ ] | Stag: 10 Ovfit: 0\n[12/11/2015 20:21:15] 7/10 | Batch 51/124 | D 1.502809e-003 [↓ ] | Valid D 1.913079e-003 [↑ ] | Stag: 20 Ovfit: 0\n[12/11/2015 20:21:15] 7/10 | Batch 61/124 | D 1.241405e-002 [↑ ] | Valid D 1.924762e-003 [↑ ] | Stag: 30 Ovfit: 0\n[12/11/2015 20:21:15] 7/10 | Batch 71/124 | D 2.962820e-003 [↓ ] | Valid D 2.024504e-003 [↑ ] | Stag: 40 Ovfit: 0\n[12/11/2015 20:21:15] 7/10 | Batch 81/124 | D 1.421725e-003 [↓ ] | Valid D 1.919308e-003 [↓ ] | Stag: 50 Ovfit: 0\n[12/11/2015 20:21:15] 7/10 | Batch 91/124 | D 3.717377e-003 [↑ ] | Valid D 1.854433e-003 [↓ ] | Stag: 60 Ovfit: 0\n[12/11/2015 20:21:15] 7/10 | Batch 101/124 | D 1.973184e-003 [↓ ] | Valid D 1.907719e-003 [↑ ] | Stag: 70 Ovfit: 0\n[12/11/2015 20:21:15] 7/10 | Batch 111/124 | D 1.190252e-002 [↑ ] | Valid D 1.867085e-003 [↓ ] | Stag: 80 Ovfit: 0\n[12/11/2015 20:21:15] 7/10 | Batch 121/124 | D 2.006255e-003 [↓ ] | Valid D 1.894716e-003 [↑ ] | Stag: 90 Ovfit: 0\n[12/11/2015 20:21:15] 8/10 | Batch 1/124 | D 1.721533e-003 [↓ ] | Valid D 1.898627e-003 [↑ ] | Stag:100 Ovfit: 0\n[12/11/2015 20:21:15] 8/10 | Batch 11/124 | D 1.553262e-003 [↓ ] | Valid D 1.897926e-003 [↓ ] | Stag:110 Ovfit: 0\n[12/11/2015 20:21:15] 8/10 | Batch 21/124 | D 5.004487e-003 [↑ ] | Valid D 1.823838e-003 [↓ ] | Stag:120 Ovfit: 0\n[12/11/2015 20:21:15] 8/10 | Batch 31/124 | D 3.308986e-003 [↓ ] | Valid D 1.768821e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:15] 8/10 | Batch 41/124 | D 6.563510e-003 [↑ ] | Valid D 1.835302e-003 [↑ ] | Stag: 10 Ovfit: 0\n[12/11/2015 20:21:15] 8/10 | Batch 51/124 | D 1.507999e-003 [↓ ] | Valid D 1.868091e-003 [↑ ] | Stag: 20 Ovfit: 0\n[12/11/2015 20:21:15] 8/10 | Batch 61/124 | D 1.148601e-002 [↑ ] | Valid D 1.876653e-003 [↑ ] | Stag: 30 Ovfit: 0\n[12/11/2015 20:21:16] 8/10 | Batch 71/124 | D 2.807777e-003 [↓ ] | Valid D 1.968064e-003 [↑ ] | Stag: 40 Ovfit: 0\n[12/11/2015 20:21:16] 8/10 | Batch 81/124 | D 1.440011e-003 [↓ ] | Valid D 1.876611e-003 [↓ ] | Stag: 50 Ovfit: 0\n[12/11/2015 20:21:16] 8/10 | Batch 91/124 | D 3.522004e-003 [↑ ] | Valid D 1.821817e-003 [↓ ] | Stag: 60 Ovfit: 0\n[12/11/2015 20:21:16] 8/10 | Batch 101/124 | D 2.031282e-003 [↓ ] | Valid D 1.872902e-003 [↑ ] | Stag: 70 Ovfit: 0\n[12/11/2015 20:21:16] 8/10 | Batch 111/124 | D 1.182362e-002 [↑ ] | Valid D 1.836957e-003 [↓ ] | Stag: 80 Ovfit: 0\n[12/11/2015 20:21:16] 8/10 | Batch 121/124 | D 2.035742e-003 [↓ ] | Valid D 1.864137e-003 [↑ ] | Stag: 90 Ovfit: 0\n[12/11/2015 20:21:16] 9/10 | Batch 1/124 | D 1.699795e-003 [↓ ] | Valid D 1.865989e-003 [↑ ] | Stag:100 Ovfit: 0\n[12/11/2015 20:21:16] 9/10 | Batch 11/124 | D 1.556397e-003 [↓ ] | Valid D 1.866347e-003 [↑ ] | Stag:110 Ovfit: 0\n[12/11/2015 20:21:16] 9/10 | Batch 21/124 | D 4.788828e-003 [↑ ] | Valid D 1.804229e-003 [↓ ] | Stag:120 Ovfit: 0\n[12/11/2015 20:21:16] 9/10 | Batch 31/124 | D 3.119682e-003 [↓ ] | Valid D 1.756223e-003 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 20:21:16] 9/10 | Batch 41/124 | D 6.336636e-003 [↑ ] | Valid D 1.816257e-003 [↑ ] | Stag: 10 Ovfit: 0\n[12/11/2015 20:21:16] 9/10 | Batch 51/124 | D 1.516153e-003 [↓ ] | Valid D 1.843593e-003 [↑ ] | Stag: 20 Ovfit: 0\n[12/11/2015 20:21:16] 9/10 | Batch 61/124 | D 1.080968e-002 [↑ ] | Valid D 1.850113e-003 [↑ ] | Stag: 30 Ovfit: 0\n[12/11/2015 20:21:16] 9/10 | Batch 71/124 | D 2.720124e-003 [↓ ] | Valid D 1.934669e-003 [↑ ] | Stag: 40 Ovfit: 0\n[12/11/2015 20:21:16] 9/10 | Batch 81/124 | D 1.455176e-003 [↓ ] | Valid D 1.852409e-003 [↓ ] | Stag: 50 Ovfit: 0\n[12/11/2015 20:21:16] 9/10 | Batch 91/124 | D 3.375944e-003 [↑ ] | Valid D 1.804057e-003 [↓ ] | Stag: 60 Ovfit: 0\n[12/11/2015 20:21:16] 9/10 | Batch 101/124 | D 2.093168e-003 [↓ ] | Valid D 1.853583e-003 [↑ ] | Stag: 70 Ovfit: 0\n[12/11/2015 20:21:16] 9/10 | Batch 111/124 | D 1.178356e-002 [↑ ] | Valid D 1.820183e-003 [↓ ] | Stag: 80 Ovfit: 0\n[12/11/2015 20:21:16] 9/10 | Batch 121/124 | D 2.061530e-003 [↓ ] | Valid D 1.846045e-003 [↑ ] | Stag: 90 Ovfit: 0\n[12/11/2015 20:21:16] 10/10 | Batch 1/124 | D 1.689459e-003 [↓ ] | Valid D 1.846794e-003 [↑ ] | Stag:100 Ovfit: 0\n[12/11/2015 20:21:16] 10/10 | Batch 11/124 | D 1.560583e-003 [↓ ] | Valid D 1.847311e-003 [↑ ] | Stag:110 Ovfit: 0\n[12/11/2015 20:21:16] 10/10 | Batch 21/124 | D 4.588457e-003 [↑ ] | Valid D 1.792883e-003 [↓ ] | Stag:120 Ovfit: 0\n[12/11/2015 20:21:16] 10/10 | Batch 31/124 | D 3.001853e-003 [↓ ] | Valid D 1.750141e-003 [↓ ] | Stag:130 Ovfit: 0\n[12/11/2015 20:21:16] 10/10 | Batch 41/124 | D 6.195725e-003 [↑ ] | Valid D 1.805622e-003 [↑ ] | Stag:140 Ovfit: 0\n[12/11/2015 20:21:16] 10/10 | Batch 51/124 | D 1.524289e-003 [↓ ] | Valid D 1.829196e-003 [↑ ] | Stag:150 Ovfit: 0\n[12/11/2015 20:21:17] 10/10 | Batch 61/124 | D 1.029841e-002 [↑ ] | Valid D 1.834366e-003 [↑ ] | Stag:160 Ovfit: 0\n[12/11/2015 20:21:17] 10/10 | Batch 71/124 | D 2.667856e-003 [↓ ] | Valid D 1.913492e-003 [↑ ] | Stag:170 Ovfit: 0\n[12/11/2015 20:21:17] 10/10 | Batch 81/124 | D 1.467351e-003 [↓ ] | Valid D 1.837669e-003 [↓ ] | Stag:180 Ovfit: 0\n[12/11/2015 20:21:17] 10/10 | Batch 91/124 | D 3.261143e-003 [↑ ] | Valid D 1.793646e-003 [↓ ] | Stag:190 Ovfit: 0\n[12/11/2015 20:21:17] 10/10 | Batch 101/124 | D 2.153974e-003 [↓ ] | Valid D 1.842048e-003 [↑ ] | Stag:200 Ovfit: 0\n[12/11/2015 20:21:17] 10/10 | Batch 111/124 | D 1.176465e-002 [↑ ] | Valid D 1.810117e-003 [↓ ] | Stag:210 Ovfit: 0\n[12/11/2015 20:21:17] 10/10 | Batch 121/124 | D 2.082179e-003 [↓ ] | Valid D 1.834467e-003 [↑ ] | Stag:220 Ovfit: 0\n[12/11/2015 20:21:17] Duration : 00:00:05.2093910\n[12/11/2015 20:21:17] Loss initial : D 4.748471e-001\n[12/11/2015 20:21:17] Loss final : D 1.395002e-003 (Best)\n[12/11/2015 20:21:17] Loss change : D -4.734521e-001 (-99.71 %)\n[12/11/2015 20:21:17] Loss chg. / s : D -9.088434e-002\n[12/11/2015 20:21:17] Epochs / s : 1.919610181\n[12/11/2015 20:21:17] Epochs / min : 115.1766109\n[12/11/2015 20:21:17] --- Training finished\n\n\n\nAfter a 5-second training, we can see that the characteristics of the problem domain (distinguishing between the digits of 0 and 1) is captured in the model weights.\n\n*)\n\nlet l = (n.[0] :?> Linear)\nl.VisualizeWRowsAsImageGrid(28) |> printfn \"%s\"\n\n(**\n [lang=cs]\n Hype.Neural.Linear\n 784 -> 1\n Learnable parameters: 785\n Init: Standard\n W's rows reshaped to (28 x 28), presented in a (1 x 1) grid:\n DM : 28 x 28\n ----------------------------\n ----------------------------\n ------------▴▴▴▴▴-----------\n ---------▴--▴▴▴▴▴▴-▴--------\n --------▴▴▪▴▪▪▪▴-▴▴▴▪▪▪▴----\n ------▴-▴▴▴▴▪▪▴▴-·-▴▪▪▪▪▴---\n ------▴--▴▴-▴▴--···▴▪▴▴▴▴---\n ----▴---------▴▴-· ---------\n ---------··---▴▪--·-·····---\n -------······▴▪▪▪▴-······---\n ------····· ▴●●●▴· ·---\n -----·· · ▪♦■♦▪ ·---\n -----· · ●■■♦▴ ·---\n -----· ·♦██♦· ·---\n -----· ▴■██● ·---\n ----· ▪██■▪ ·---\n ----· -●█■♦- ·---\n ----· ▴♦█■●· ···---\n ----· ·▴♦♦♦●· ····----\n ----· ·▴▪●●●▪·· ····--▴--\n ----······-▴▪▪▪▴--------▴---\n -----▴▴----·--▴▴-▴▴-▴▴------\n -----▴▪▪▴-· ··--▴▴▪▴▴▴▴-----\n ----▴▪▪▪▪▴-· ·-▴▴▪▴▴▴▴------\n -----▴▪▴▴▴▴·---▴▴▴▴▴--------\n ------------▴▴▴▴------------\n ----------------------------\n ----------------------------\n\n b:\n DV : 1\n \n### Classifier\n\nYou can create classifiers by instantiating types such as **LogisticClassifier** or **SoftmaxClassifier**, and passing a classification function of the form **DM->DM**in the constructor. Alternatively, you can directly pass the model we have just trained. \n\nPlease see the [API reference](reference/index.html) and the [source code](https://github.com/hypelib/Hype/blob/master/src/Hype/Classifier.fs) for a better understanding of how classifiers are implemented.\n\n*)\n\nlet cc = LogisticClassifier(n)\n\n(**\nLet's test the class predictions for 10 random elements from the MNIST test set, which, if you remember, we've filtered to have only 0s and 1s.\n*)\n\nlet pred = cc.Classify(MNISTtest01.X.[*,0..9]);;\nlet real = MNISTtest01.Y.[*, 0..9] |> DM.toDV |> DV.toArray |> Array.map (float32>>int)\n\n(**\n
\nval pred : int [] = [|1; 0; 1; 0; 1; 0; 0; 1; 1; 1|]\nval real : int [] = [|1; 0; 1; 0; 1; 0; 0; 1; 1; 1|]\n\n\nThe classifier seems to be working well. We can compute the classification error for a given dataset.\n*)\n\nlet error = cc.ClassificationError(MNISTtest01);;\n\n(**\n
\nval error : float32 = 0.000472813234f\n\n\nThe classification error is 0.047%.\n\nFinally, this is how you would classify single digits.\n*)\n\nlet cls = cc.Classify(MNISTtest01.X.[*,0]);;\nMNISTtest01.X.[*,0] |> DV.visualizeAsDM 28 |> printfn \"%s\"\n\n(**\n [lang=cs]\n val cls : int = 1\n\n DM : 28 x 28\n \n \n \n \n ♦ \n ●♦ \n █ \n ■· \n ▪█- \n ▴█- \n ■♦ \n ♦█· \n -█▪ \n █▪ \n ●▪ \n ▪█ \n ▪█- \n ▪█▴ \n ▪█■ \n █■ \n ██ \n ▪█ \n █▴ \n █● \n \n \nAnd this is how you would classify many digits efficiently at the same time, by running them through the model together as the columns of an input matrix.\n*)\n\nlet clss = cc.Classify(MNISTtest01.X.[*,5..9]);;\nMNISTtest01.[5..9].VisualizeXColsAsImageGrid(28) |> printfn \"%s\"\n\n(**\n [lang=cs]\n val clss : int [] = [|0; 0; 1; 1; 1|]\n\n Hype.Dataset\n X: 784 x 5\n Y: 1 x 5\n X's columns reshaped to (28 x 28), presented in a (2 x 3) grid:\n DM : 56 x 84\n \n \n \n \n ██· \n ●███♦- ·████♦· -█▴ \n ██████■- ♦███████- ▪██· \n ●███████■ ♦███████████▴ ●██· \n ▪███● -███- ♦█████████████■ ▴♦█· \n ♦██▪ -██■ ████♦ ●●████████ ▪█· \n ▪███ ·██■ ●█████· ··██████ ▪█· \n ■██▪ ·██■ ▪██████· ██████ ▪█· \n ·██■ ▪██■ -██████♦ ██████ ▪█· \n ♦██▪ ■██■ ▪███████ ·█████♦ ●█· \n ·███- ■██■ ██████♦· ♦██████· ██· \n ♦██■ ·███■ ██████- ▪███████- ██· \n ■██- -███- ██████ ●███████▴ ██· \n ■██· ■██♦ ██████·♦████████■ ██· \n ■██· ■███● ████████████████ ██ \n ■██● ████ ██████████████- ██ \n ▴███· -■███- ▴████████████▴ ·██ \n ·█████████♦ ■████████■● ██ \n ▴███████♦ ████████ ·█♦· \n ▪███♦· ●●■●●- ·██▴ \n █■ \n \n \n \n \n \n \n \n ·██ \n ███ -██■ \n ●██■ ▪████- \n ♦██ ♦███■ \n ·███ ·████■ \n ■██♦ ▴███■ \n ▪██■ -████● \n ███- ▴████▪ \n ▪██■ ·████♦ \n -███▴ █████· \n ♦██● ████♦ \n -███ ▪████· \n ███▴ ■███■ \n -███ ▴████· \n ███● ■███■ \n ████ ■███▪ \n ●███- -███♦ \n ▴███● ●███▪ \n ●██♦ ████▪ \n ●██· ■███▴ \n ■■- \n \n \n \n \n*)\n" }, { "path": "docs/input/Training.fsx", "content": "(*** hide ***)\n#r \"../../src/Hype/bin/Release/netstandard2.0/DiffSharp.dll\"\n#r \"../../src/Hype/bin/Release/netstandard2.0/Hype.dll\"\n#I \"../../packages/R.NET.Community/lib/net40/\"\n#I \"../../packages/R.NET.Community.FSharp/lib/net40/\"\n#I \"../../packages/RProvider\"\n#load \"RProvider.fsx\"\n//fsi.ShowDeclarationValues <- false\nSystem.Environment.CurrentDirectory <- __SOURCE_DIRECTORY__\n\n(**\nTraining\n========\n\nIn [optimization,](optimization.html) we've seen how nested AD and gradient-based optimization work together.\n\nTraining a model is the optimization of model parameters to minimize a loss function, or equivalently, to maximize the likelihood of a given set of data under the model parameters. In addition to the _optimization method_, _learning rate_, _momentum_, and _gradient clipping_ parameters we've seen, this introduces parameters for the _loss function_, _regularization_, _training batches_, and _validation and early stopping_. \n\nBut let's start with the **Dataset** type, which we will use for keeping the training, validation, and test data for the training procedure.\n\nDataset\n-------\n\nFor supervised training, data consists of pairs of input vectors $\\mathbf{x}_i \\in \\mathbb{R}^{d_x}$ and output vectors $\\mathbf{y}_i \\in \\mathbb{R}^{d_y}$. We represent data using the **Dataset** type, which is basically a pair of matrices \n\n$$$\n \\begin{eqnarray*}\n \\mathbf{X} &\\in& \\mathbb{R}^{d_x \\times n}\\\\\n \\mathbf{Y} &\\in& \\mathbb{R}^{d_y \\times n}\\\\\n \\end{eqnarray*}\n \nholding these vectors, where $n$ is the number of input–output pairs, $d_x$ is the number of input features and $d_y$ is the number of output features. In other words, each of the $n$ columns of the matrix $\\mathbf{X}$ is an input vector of length $d_x$ and each of the $n$ columns of matrix $\\mathbf{Y}$ is the corresponding output vector of length $d_y$.\n\nKeeping data in matrix form is essential for harnessing high-performance linear algebra engines tailored for your CPU or GPU. Hype, by default, uses a high-performance CPU backend using OpenBLAS for BLAS/LAPACK operations, and parallel implementations of non-BLAS operations such as elementwise functions.\n*)\n\nopen Hype\nopen DiffSharp.AD.Float32\n\nlet x = toDM [[0; 0; 1; 1]\n [0; 1; 0; 1]]\nlet y = toDM [[0; 1; 1; 0]]\n\nlet XORdata = Dataset(x, y)\n\n(**\n\nHype provides several utility functions for loading data into matrices from images, delimited text files (e.g., CSV), or commonly used dataset files such as the MNIST.\n\n*)\n\nlet MNIST = Dataset(Util.LoadMNISTPixels(\"train-images.idx3-ubyte\", 60000),\n Util.LoadMNISTLabels(\"train-labels.idx1-ubyte\", 60000) |> toDV |> DM.ofDV 1).NormalizeX()\n\nlet MNISTtest = Dataset(Util.LoadMNISTPixels(\"t10k-images.idx3-ubyte\", 10000),\n Util.LoadMNISTLabels(\"t10k-labels.idx1-ubyte\", 10000) |> toDV |> DM.ofDV 1).NormalizeX()\n\n(**\n\nYou can see the [API reference](reference/index.html) and the [source code](https://github.com/hypelib/Hype/blob/master/src/Hype/Hype.fs) for various ways of constructing Datasets.\n\nTraining parameters\n-------------------\n\nLet's load the housing prices dataset from the [Stanford UFLDL Tutorial](http://ufldl.stanford.edu/tutorial/supervised/LinearRegression/) and divide it into input and output pairs. We will later train a simple linear regression model, to demonstrate the use of training parameters.\n\n*)\n\nlet h = Util.LoadDelimited(\"housing.data\") |> DM.Transpose\nh.ToString() |> printfn \"%s\"\n\n(**\n
\nDM : 14 x 506\n 0.00632 0.0273 0.0273 0.0324 0.0691 0.0299 0.0883 0.145 0.211 0.17 0.225 0.117 0.0938 0.63 0.638 0.627 1.05 0.784 0.803 0.726 1.25 0.852 1.23 0.988 0.75 0.841 0.672 0.956 0.773 1 1.13 1.35 1.39 1.15 1.61 0.0642 0.0974 0.0801 0.175 0.0276 0.0336 0.127 0.142 0.159 0.123 0.171 0.188 0.229 0.254 0.22 0.0887 0.0434 0.0536 0.0498 0.0136 0.0131 0.0206 0.0143 0.154 0.103 0.149 0.172 0.11 0.127 0.0195 0.0358 0.0438 0.0579 0.136 0.128 0.0883 0.159 0.0916 0.195 0.079 0.0951 0.102 0.0871 0.0565 0.0839 0.0411 0.0446 0.0366 0.0355 0.0506 0.0574 0.0519 0.0715 0.0566 0.053 0.0468 0.0393 0.042 0.0288 0.0429 0.122 0.115 0.121 0.0819 0.0686 0.149 0.114 0.229 0.212 0.14 0.133 0.171 0.131 0.128 0.264 0.108 0.101 0.123 0.222 0.142 0.171 0.132 0.151 0.131 0.145 0.069 0.0717 0.093 0.15 0.0985 0.169 0.387 0.259 0.325 0.881 0.34 1.19 0.59 0.33 0.976 0.558 0.323 0.352 0.25 0.545 0.291 1.63 3.32 4.1 2.78 2.38 2.16 2.37 2.33 2.73 1.66 1.5 1.13 2.15 1.41 3.54 2.45 1.22 1.34 1.43 1.27 1.46 1.83 1.52 2.24 2.92 2.01 1.8 2.3 2.45 1.21 2.31 0.139 0.0918 0.0845 0.0666 0.0702 0.0543 0.0664 0.0578 0.0659 0.0689 0.091 0.1 0.0831 0.0605 0.056 0.0788 0.126 0.0837 0.0907 0.0691 0.0866 0.0219 0.0144 0.0138 0.0401 0.0467 0.0377 0.0315 0.0178 0.0345 0.0218 0.0351 0.0201 0.136 0.23 0.252 0.136 0.436 0.174 0.376 0.217 0.141 0.29 0.198 0.0456 0.0701 0.111 0.114 0.358 0.408 0.624 0.615 0.315 0.527 0.382 0.412 0.298 0.442 0.537 0.463 0.575 0.331 0.448 0.33 0.521 0.512 0.0824 0.0925 0.113 0.106 0.103 0.128 0.206 0.191 0.34 0.197 0.164 0.191 0.14 0.214 0.0822 0.369 0.0482 0.0355 0.0154 0.612 0.664 0.657 0.54 0.534 0.52 0.825 0.55 0.762 0.786 0.578 0.541 0.0907 0.299 0.162 0.115 0.222 0.0564 0.096 0.105 0.0613 0.0798 0.21 0.0358 0.0371 0.0613 0.015 0.00906 0.011 0.0197 0.0387 0.0459 0.043 0.035 0.0789 0.0362 0.0827 0.082 0.129 0.0537 0.141 0.0647 0.0556 0.0442 0.0354 0.0927 0.1 0.0552 0.0548 0.075 0.0493 0.493 0.349 2.64 0.79 0.262 0.269 0.369 0.254 0.318 0.245 0.402 0.475 0.168 0.182 0.351 0.284 0.341 0.192 0.303 0.241 0.0662 0.0672 0.0454 0.0502 0.0347 0.0508 0.0374 0.0396 0.0343 0.0304 0.0331 0.055 0.0615 0.013 0.025 0.0254 0.0305 0.0311 0.0616 0.0187 0.015 0.029 0.0621 0.0795 0.0724 0.0171 0.043 0.107 8.98 3.85 5.2 4.26 4.54 3.84 3.68 4.22 3.47 4.56 3.7 3.52 4.9 5.67 6.54 9.23 8.27 1.11 8.5 9.61 5.29 9.82 3.65 7.87 8.98 5.87 9.19 7.99 0.0849 6.81 4.39 2.6 4.33 8.15 6.96 5.29 1.58 8.64 3.36 8.72 5.87 7.67 8.35 9.92 5.05 4.24 9.6 4.8 1.53 7.92 0.716 1.95 7.4 4.44 1.14 4.05 8.81 8.66 5.75 8.08 0.834 5.94 3.53 1.81 1.09 7.02 2.05 7.05 8.79 5.86 2.25 7.66 7.37 9.34 8.49 0.0623 6.44 5.58 3.91 1.16 4.42 5.18 3.68 9.39 2.05 9.72 5.67 9.97 2.8 0.672 6.29 9.92 9.33 7.53 6.72 5.44 5.09 8.25 9.51 4.75 4.67 8.2 7.75 6.8 4.81 3.69 6.65 5.82 7.84 3.16 3.77 4.42 5.58 3.08 4.35 4.04 3.57 4.65 8.06 6.39 4.87 5.02 0.233 4.33 5.82 5.71 5.73 2.82 2.38 3.67 5.69 4.84 0.151 0.183 0.207 0.106 0.111 0.173 0.28 0.179 0.29 0.268 0.239 0.178 0.224 0.0626 0.0453 0.0608 0.11 0.0474 \n 18 0 0 0 0 0 12.5 12.5 12.5 12.5 12.5 12.5 12.5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 75 75 0 0 0 0 0 0 0 0 0 21 21 21 21 75 90 85 100 25 25 25 25 25 25 17.5 80 80 12.5 12.5 12.5 0 0 0 0 0 0 0 0 0 0 25 25 25 25 0 0 0 0 0 0 0 0 28 28 28 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 45 45 45 45 45 45 60 60 80 80 80 80 95 95 82.5 82.5 95 95 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 30 30 30 30 30 30 22 22 22 22 22 22 22 22 22 22 80 80 90 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 40 40 40 40 40 20 20 20 20 90 90 55 80 52.5 52.5 52.5 80 80 80 0 0 0 0 0 70 70 70 34 34 34 33 33 33 33 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 35 35 0 0 0 0 0 0 0 0 35 0 55 55 0 0 85 80 40 40 60 60 90 80 80 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 \n 2.31 7.07 7.07 2.18 2.18 2.18 7.87 7.87 7.87 7.87 7.87 7.87 7.87 8.14 8.14 8.14 8.14 8.14 8.14 8.14 8.14 8.14 8.14 8.14 8.14 8.14 8.14 8.14 8.14 8.14 8.14 8.14 8.14 8.14 8.14 5.96 5.96 5.96 5.96 2.95 2.95 6.91 6.91 6.91 6.91 6.91 6.91 6.91 6.91 6.91 5.64 5.64 5.64 5.64 4 1.22 0.74 1.32 5.13 5.13 5.13 5.13 5.13 5.13 1.38 3.37 3.37 6.07 6.07 6.07 10.8 10.8 10.8 10.8 12.8 12.8 12.8 12.8 12.8 12.8 4.86 4.86 4.86 4.86 4.49 4.49 4.49 4.49 3.41 3.41 3.41 3.41 15 15 15 2.89 2.89 2.89 2.89 2.89 8.56 8.56 8.56 8.56 8.56 8.56 8.56 8.56 8.56 8.56 8.56 10 10 10 10 10 10 10 10 10 25.7 25.7 25.7 25.7 25.7 25.7 25.7 21.9 21.9 21.9 21.9 21.9 21.9 21.9 21.9 21.9 21.9 21.9 21.9 21.9 21.9 21.9 19.6 19.6 19.6 19.6 19.6 19.6 19.6 19.6 19.6 19.6 19.6 19.6 19.6 19.6 19.6 19.6 19.6 19.6 19.6 19.6 19.6 19.6 19.6 19.6 19.6 19.6 19.6 19.6 19.6 19.6 4.05 4.05 4.05 4.05 4.05 4.05 4.05 2.46 2.46 2.46 2.46 2.46 2.46 2.46 2.46 3.44 3.44 3.44 3.44 3.44 3.44 2.93 2.93 0.46 1.52 1.52 1.52 1.47 1.47 2.03 2.03 2.68 2.68 10.6 10.6 10.6 10.6 10.6 10.6 10.6 10.6 10.6 10.6 10.6 13.9 13.9 13.9 13.9 6.2 6.2 6.2 6.2 6.2 6.2 6.2 6.2 6.2 6.2 6.2 6.2 6.2 6.2 6.2 6.2 6.2 6.2 4.93 4.93 4.93 4.93 4.93 4.93 5.86 5.86 5.86 5.86 5.86 5.86 5.86 5.86 5.86 5.86 3.64 3.64 3.75 3.97 3.97 3.97 3.97 3.97 3.97 3.97 3.97 3.97 3.97 3.97 3.97 6.96 6.96 6.96 6.96 6.96 6.41 6.41 6.41 6.41 6.41 3.33 3.33 3.33 3.33 1.21 2.97 2.25 1.76 5.32 5.32 5.32 4.95 4.95 4.95 13.9 13.9 13.9 13.9 13.9 2.24 2.24 2.24 6.09 6.09 6.09 2.18 2.18 2.18 2.18 9.9 9.9 9.9 9.9 9.9 9.9 9.9 9.9 9.9 9.9 9.9 9.9 7.38 7.38 7.38 7.38 7.38 7.38 7.38 7.38 3.24 3.24 3.24 6.06 6.06 5.19 5.19 5.19 5.19 5.19 5.19 5.19 5.19 1.52 1.89 3.78 3.78 4.39 4.39 4.15 2.01 1.25 1.25 1.69 1.69 2.02 1.91 1.91 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 18.1 27.7 27.7 27.7 27.7 27.7 9.69 9.69 9.69 9.69 9.69 9.69 9.69 9.69 11.9 11.9 11.9 11.9 11.9 \n 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 1 1 0 0 0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 1 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 1 0 1 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 1 1 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 \n 0.538 0.469 0.469 0.458 0.458 0.458 0.524 0.524 0.524 0.524 0.524 0.524 0.524 0.538 0.538 0.538 0.538 0.538 0.538 0.538 0.538 0.538 0.538 0.538 0.538 0.538 0.538 0.538 0.538 0.538 0.538 0.538 0.538 0.538 0.538 0.499 0.499 0.499 0.499 0.428 0.428 0.448 0.448 0.448 0.448 0.448 0.448 0.448 0.448 0.448 0.439 0.439 0.439 0.439 0.41 0.403 0.41 0.411 0.453 0.453 0.453 0.453 0.453 0.453 0.416 0.398 0.398 0.409 0.409 0.409 0.413 0.413 0.413 0.413 0.437 0.437 0.437 0.437 0.437 0.437 0.426 0.426 0.426 0.426 0.449 0.449 0.449 0.449 0.489 0.489 0.489 0.489 0.464 0.464 0.464 0.445 0.445 0.445 0.445 0.445 0.52 0.52 0.52 0.52 0.52 0.52 0.52 0.52 0.52 0.52 0.52 0.547 0.547 0.547 0.547 0.547 0.547 0.547 0.547 0.547 0.581 0.581 0.581 0.581 0.581 0.581 0.581 0.624 0.624 0.624 0.624 0.624 0.624 0.624 0.624 0.624 0.624 0.624 0.624 0.624 0.624 0.624 0.871 0.871 0.871 0.871 0.871 0.871 0.871 0.871 0.871 0.871 0.871 0.871 0.871 0.871 0.871 0.605 0.605 0.871 0.605 0.605 0.605 0.605 0.605 0.605 0.605 0.605 0.605 0.605 0.605 0.605 0.51 0.51 0.51 0.51 0.51 0.51 0.51 0.488 0.488 0.488 0.488 0.488 0.488 0.488 0.488 0.437 0.437 0.437 0.437 0.437 0.437 0.401 0.401 0.422 0.404 0.404 0.404 0.403 0.403 0.415 0.415 0.416 0.416 0.489 0.489 0.489 0.489 0.489 0.489 0.489 0.489 0.489 0.489 0.489 0.55 0.55 0.55 0.55 0.507 0.507 0.507 0.507 0.504 0.504 0.504 0.504 0.504 0.504 0.504 0.504 0.507 0.507 0.507 0.507 0.507 0.507 0.428 0.428 0.428 0.428 0.428 0.428 0.431 0.431 0.431 0.431 0.431 0.431 0.431 0.431 0.431 0.431 0.392 0.392 0.394 0.647 0.647 0.647 0.647 0.647 0.647 0.647 0.647 0.647 0.647 0.575 0.575 0.464 0.464 0.464 0.464 0.464 0.447 0.447 0.447 0.447 0.447 0.443 0.443 0.443 0.443 0.401 0.4 0.389 0.385 0.405 0.405 0.405 0.411 0.411 0.411 0.437 0.437 0.437 0.437 0.437 0.4 0.4 0.4 0.433 0.433 0.433 0.472 0.472 0.472 0.472 0.544 0.544 0.544 0.544 0.544 0.544 0.544 0.544 0.544 0.544 0.544 0.544 0.493 0.493 0.493 0.493 0.493 0.493 0.493 0.493 0.46 0.46 0.46 0.438 0.438 0.515 0.515 0.515 0.515 0.515 0.515 0.515 0.515 0.442 0.518 0.484 0.484 0.442 0.442 0.429 0.435 0.429 0.429 0.411 0.411 0.41 0.413 0.413 0.77 0.77 0.77 0.77 0.77 0.77 0.77 0.77 0.718 0.718 0.718 0.631 0.631 0.631 0.631 0.631 0.668 0.668 0.668 0.671 0.671 0.671 0.671 0.671 0.671 0.671 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.693 0.693 0.693 0.693 0.693 0.693 0.693 0.693 0.693 0.693 0.693 0.693 0.693 0.659 0.659 0.597 0.597 0.597 0.597 0.597 0.597 0.693 0.679 0.679 0.679 0.679 0.718 0.718 0.718 0.614 0.614 0.584 0.679 0.584 0.679 0.679 0.679 0.584 0.584 0.584 0.713 0.713 0.74 0.74 0.74 0.74 0.74 0.74 0.74 0.74 0.74 0.74 0.74 0.74 0.74 0.713 0.713 0.713 0.713 0.713 0.713 0.713 0.713 0.713 0.713 0.713 0.713 0.713 0.713 0.713 0.713 0.655 0.655 0.655 0.584 0.58 0.58 0.58 0.532 0.58 0.614 0.584 0.584 0.614 0.614 0.614 0.614 0.532 0.532 0.532 0.532 0.583 0.583 0.583 0.583 0.609 0.609 0.609 0.609 0.609 0.585 0.585 0.585 0.585 0.585 0.585 0.585 0.585 0.573 0.573 0.573 0.573 0.573 \n 6.58 6.42 7.19 7 7.15 6.43 6.01 6.17 5.63 6 6.38 6.01 5.89 5.95 6.1 5.83 5.94 5.99 5.46 5.73 5.57 5.97 6.14 5.81 5.92 5.6 5.81 6.05 6.5 6.67 5.71 6.07 5.95 5.7 6.1 5.93 5.84 5.85 5.97 6.6 7.02 6.77 6.17 6.21 6.07 5.68 5.79 6.03 5.4 5.6 5.96 6.12 6.51 6 5.89 7.25 6.38 6.82 6.15 5.93 5.74 5.97 6.46 6.76 7.1 6.29 5.79 5.88 5.59 5.89 6.42 5.96 6.07 6.25 6.27 6.29 6.28 6.14 6.23 5.87 6.73 6.62 6.3 6.17 6.39 6.63 6.02 6.12 7.01 7.08 6.42 6.41 6.44 6.21 6.25 6.63 6.16 8.07 7.82 7.42 6.73 6.78 6.41 6.14 6.17 5.85 5.84 6.13 6.47 6.23 6.2 6.72 5.91 6.09 6.25 5.93 6.18 6.02 5.87 5.73 5.87 6 5.96 5.86 5.88 5.99 5.61 5.69 6.43 5.64 6.46 6.33 6.37 5.82 5.76 6.34 5.94 6.45 5.86 6.15 6.17 5.02 5.4 5.47 4.9 6.13 5.63 4.93 5.19 5.6 6.12 5.4 5.01 5.71 6.13 6.15 5.27 6.94 6.07 6.51 6.25 7.49 7.8 8.38 5.85 6.1 7.93 5.88 6.32 6.4 5.88 5.88 5.57 6.42 5.86 6.55 6.02 6.32 6.86 6.98 7.77 6.14 7.16 6.56 5.6 6.15 7.83 6.78 6.56 7.19 6.95 6.74 7.18 6.8 6.6 7.88 7.29 7.11 7.27 6.98 7.14 6.16 7.61 7.85 8.03 5.89 6.33 5.78 6.06 5.34 5.96 5.4 5.81 6.38 5.41 6.18 5.89 6.64 5.95 6.37 6.95 6.16 6.88 6.62 8.27 8.73 8.04 7.16 7.69 6.55 5.98 7.41 8.34 8.25 6.73 6.09 6.63 7.36 6.48 6.61 6.9 6.1 6.36 6.39 5.59 5.61 6.11 6.23 6.43 6.72 6.49 6.44 6.96 8.26 6.11 5.88 7.45 8.7 7.33 6.84 7.2 7.52 8.4 7.33 7.21 5.56 7.01 8.3 7.47 5.92 5.86 6.24 6.54 7.69 6.76 6.85 7.27 6.83 6.48 6.81 7.82 6.97 7.65 7.92 7.09 6.45 6.23 6.21 6.32 6.57 6.86 7.15 6.63 6.13 6.01 6.68 6.55 5.79 6.35 7.04 6.87 6.59 6.5 6.98 7.24 6.62 7.42 6.85 6.64 5.97 4.97 6.12 6.02 6.27 6.57 5.71 5.91 5.78 6.38 6.11 6.43 6.38 6.04 5.71 6.42 6.43 6.31 6.08 5.87 6.33 6.14 5.71 6.03 6.32 6.31 6.04 5.87 5.9 6.06 5.99 5.97 7.24 6.54 6.7 6.87 6.01 5.9 6.52 6.64 6.94 6.49 6.58 5.88 6.73 5.66 5.94 6.21 6.4 6.13 6.11 6.4 6.25 5.36 5.8 8.78 3.56 4.96 3.86 4.97 6.68 7.02 6.22 5.88 4.91 4.14 7.31 6.65 6.79 6.38 6.22 6.97 6.55 5.54 5.52 4.37 5.28 4.65 5 4.88 5.39 5.71 6.05 5.04 6.19 5.89 6.47 6.41 5.75 5.45 5.85 5.99 6.34 6.4 5.35 5.53 5.68 4.14 5.61 5.62 6.85 5.76 6.66 4.63 5.16 4.52 6.43 6.78 5.3 5.96 6.82 6.41 6.01 5.65 6.1 5.57 5.9 5.84 6.2 6.19 6.38 6.35 6.83 6.43 6.44 6.21 6.63 6.46 6.15 5.94 5.63 5.82 6.41 6.22 6.49 5.85 6.46 6.34 6.25 6.19 6.42 6.75 6.66 6.3 7.39 6.73 6.53 5.98 5.94 6.3 6.08 6.7 6.38 6.32 6.51 6.21 5.76 5.95 6 5.93 5.71 6.17 6.23 6.44 6.98 5.43 6.16 6.48 5.3 6.19 6.23 6.24 6.75 7.06 5.76 5.87 6.31 6.11 5.91 5.45 5.41 5.09 5.98 5.98 5.71 5.93 5.67 5.39 5.79 6.02 5.57 6.03 6.59 6.12 6.98 6.79 6.03 \n 65.2 78.9 61.1 45.8 54.2 58.7 66.6 96.1 100 85.9 94.3 82.9 39 61.8 84.5 56.5 29.3 81.7 36.6 69.5 98.1 89.2 91.7 100 94.1 85.7 90.3 88.8 94.4 87.3 94.1 100 82 95 96.9 68.2 61.4 41.5 30.2 21.8 15.8 2.9 6.6 6.5 40 33.8 33.3 85.5 95.3 62 45.7 63 21.1 21.4 47.6 21.9 35.7 40.5 29.2 47.2 66.2 93.4 67.8 43.4 59.5 17.8 31.1 21.4 36.8 33 6.6 17.5 7.8 6.2 6 45 74.5 45.8 53.7 36.6 33.5 70.4 32.2 46.7 48 56.1 45.1 56.8 86.3 63.1 66.1 73.9 53.6 28.9 77.3 57.8 69.6 76 36.9 62.5 79.9 71.3 85.4 87.4 90 96.7 91.9 85.2 97.1 91.2 54.4 81.6 92.9 95.4 84.2 88.2 72.5 82.6 73.1 65.2 69.7 84.1 92.9 97 95.8 88.4 95.6 96 98.8 94.7 98.9 97.7 97.9 95.4 98.4 98.2 93.5 98.4 98.2 97.9 93.6 100 100 100 97.8 100 100 95.7 93.8 94.9 97.3 100 88 98.5 96 82.6 94 97.4 100 100 92.6 90.8 98.2 93.9 91.8 93 96.2 79.2 96.1 95.2 94.6 97.3 88.5 84.1 68.7 33.1 47.2 73.4 74.4 58.4 83.3 62.2 92.2 95.6 89.8 68.8 53.6 41.1 29.1 38.9 21.5 30.8 26.3 9.9 18.8 32 34.1 36.6 38.3 15.3 13.9 38.4 15.7 33.2 31.9 22.3 52.5 72.7 59.1 100 92.1 88.6 53.8 32.3 9.8 42.4 56 85.1 93.8 92.4 88.5 91.3 77.7 80.8 78.3 83 86.5 79.9 17 21.4 68.1 76.9 73.3 70.4 66.5 61.5 76.5 71.6 18.5 42.2 54.3 65.1 52.9 7.8 76.5 70.2 34.9 79.2 49.1 17.5 13 8.9 6.8 8.4 32 19.1 34.2 86.9 100 100 81.8 89.4 91.5 94.5 91.6 62.8 84.6 67 52.6 61.5 42.1 16.3 58.7 51.8 32.9 42.8 49 27.6 32.1 32.2 64.5 37.2 49.7 24.8 20.8 31.9 31.5 31.3 45.6 22.9 27.9 27.7 23.4 18.4 42.3 31.1 51 58 20.1 10 47.4 40.4 18.4 17.7 41.1 58.1 71.9 70.3 82.5 76.7 37.8 52.8 90.4 82.8 87.3 77.7 83.2 71.7 67.2 58.8 52.3 54.3 49.9 74.3 40.1 14.7 28.9 43.7 25.8 17.2 32.2 28.4 23.3 38.1 38.5 34.5 46.3 59.6 37.3 45.4 58.5 49.3 59.7 56.4 28.1 48.5 52.3 27.7 29.7 34.5 44.4 35.9 18.5 36.1 21.9 19.5 97.4 91 83.4 81.3 88 91.1 96.2 89 82.9 87.9 91.4 100 100 96.8 97.5 100 89.6 100 100 97.9 93.3 98.8 96.2 100 91.9 99.1 100 100 91.2 98.1 100 89.5 100 98.9 97 82.5 97 92.6 94.7 98.8 96 98.9 100 77.8 100 100 100 96 85.4 100 100 100 97.9 100 100 100 100 100 100 100 90.8 89.1 100 76.5 100 95.3 87.6 85.1 70.6 95.4 59.7 78.7 78.1 95.6 86.1 94.3 74.8 87.9 95 94.6 93.3 100 87.9 93.9 92.4 97.2 100 100 96.6 94.8 96.4 96.6 98.7 98.3 92.6 98.2 91.8 99.3 94.1 86.5 87.9 80.3 83.7 84.4 90 88.4 83 89.9 65.4 48.2 84.7 94.5 71 56.7 84 90.7 75 67.6 95.4 97.4 93.6 97.3 96.7 88 64.7 74.9 77 40.3 41.9 51.9 79.8 53.2 92.7 98.3 98 98.8 83.5 54 42.6 28.8 72.9 70.6 65.3 73.5 79.7 69.1 76.7 91 89.3 80.8 \n 4.09 4.97 4.97 6.06 6.06 6.06 5.56 5.95 6.08 6.59 6.35 6.23 5.45 4.71 4.46 4.5 4.5 4.26 3.8 3.8 3.8 4.01 3.98 4.1 4.4 4.45 4.68 4.45 4.45 4.24 4.23 4.18 3.99 3.79 3.76 3.36 3.38 3.93 3.85 5.4 5.4 5.72 5.72 5.72 5.72 5.1 5.1 5.69 5.87 6.09 6.81 6.81 6.81 6.81 7.32 8.7 9.19 8.32 7.81 6.93 7.23 6.82 7.23 7.98 9.22 6.61 6.61 6.5 6.5 6.5 5.29 5.29 5.29 5.29 4.25 4.5 4.05 4.09 5.01 4.5 5.4 5.4 5.4 5.4 4.78 4.44 4.43 3.75 3.42 3.41 3.09 3.09 3.67 3.67 3.62 3.5 3.5 3.5 3.5 3.5 2.78 2.86 2.71 2.71 2.42 2.11 2.21 2.12 2.43 2.55 2.78 2.68 2.35 2.55 2.26 2.46 2.73 2.75 2.48 2.76 2.26 2.2 2.09 1.94 2.01 1.99 1.76 1.79 1.81 1.98 2.12 2.27 2.33 2.47 2.35 2.11 1.97 1.85 1.67 1.67 1.61 1.44 1.32 1.41 1.35 1.42 1.52 1.46 1.53 1.53 1.62 1.59 1.61 1.62 1.75 1.75 1.74 1.88 1.76 1.77 1.8 1.97 2.04 2.16 2.42 2.28 2.05 2.43 2.1 2.26 2.43 2.39 2.6 2.65 2.7 3.13 3.55 3.32 2.92 2.83 2.74 2.6 2.7 2.85 2.99 3.28 3.2 3.79 4.57 4.57 6.48 6.48 6.48 6.22 6.22 5.65 7.31 7.31 7.31 7.65 7.65 6.27 6.27 5.12 5.12 3.95 4.35 4.35 4.24 3.88 3.88 3.67 3.65 3.95 3.59 3.95 3.11 3.42 2.89 3.36 2.86 3.05 3.27 3.27 2.89 2.89 3.22 3.22 3.38 3.38 3.67 3.67 3.84 3.65 3.65 3.65 4.15 4.15 6.19 6.19 6.34 6.34 7.04 7.04 7.95 7.95 8.06 8.06 7.83 7.83 7.4 7.4 8.91 8.91 9.22 9.22 6.34 1.8 1.89 2.01 2.11 2.14 2.29 2.08 1.93 1.99 2.13 2.42 2.87 3.92 4.43 4.43 3.92 4.37 4.08 4.27 4.79 4.86 4.14 4.1 4.69 5.24 5.21 5.89 7.31 7.31 9.09 7.32 7.32 7.32 5.12 5.12 5.12 5.5 5.5 5.96 5.96 6.32 7.83 7.83 7.83 5.49 5.49 5.49 4.02 3.37 3.1 3.18 3.32 3.1 2.52 2.64 2.83 3.26 3.6 3.95 4 4.03 3.53 4 4.54 4.54 4.72 4.72 4.72 5.42 5.42 5.42 5.21 5.21 5.87 6.64 6.64 6.46 6.46 5.99 5.23 5.62 4.81 4.81 4.81 7.04 6.27 5.73 6.47 8.01 8.01 8.54 8.34 8.79 8.79 10.7 10.7 12.1 10.6 10.6 2.12 2.51 2.72 2.51 2.52 2.3 2.1 1.9 1.9 1.61 1.75 1.51 1.33 1.36 1.2 1.17 1.13 1.17 1.14 1.32 1.34 1.36 1.39 1.39 1.42 1.52 1.58 1.53 1.44 1.43 1.47 1.52 1.59 1.73 1.93 2.17 1.77 1.79 1.78 1.73 1.68 1.63 1.49 1.5 1.59 1.57 1.64 1.7 1.61 1.43 1.18 1.29 1.45 1.47 1.41 1.53 1.55 1.59 1.66 1.83 1.82 1.65 1.8 1.79 1.86 1.87 1.95 2.02 2.06 1.91 2 1.86 1.94 1.97 2.05 2.09 2.2 2.32 2.22 2.12 2 1.91 1.82 1.82 1.87 2.07 2 1.98 1.9 1.99 2.07 2.2 2.26 2.19 2.32 2.36 2.37 2.45 2.5 2.44 2.58 2.78 2.78 2.72 2.6 2.57 2.73 2.8 2.96 3.07 2.87 2.54 2.91 2.82 3.03 3.1 2.9 2.53 2.43 2.21 2.31 2.1 2.17 1.95 3.42 3.33 3.41 4.1 3.72 3.99 3.55 3.15 1.82 1.76 1.82 1.87 2.11 2.38 2.38 2.8 2.8 2.89 2.41 2.4 2.5 2.48 2.29 2.17 2.39 2.51 \n 1 2 2 3 3 3 5 5 5 5 5 5 5 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 5 5 5 5 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 3 5 2 5 8 8 8 8 8 8 3 4 4 4 4 4 4 4 4 4 5 5 5 5 5 5 4 4 4 4 3 3 3 3 2 2 2 2 4 4 4 2 2 2 2 2 5 5 5 5 5 5 5 5 5 5 5 6 6 6 6 6 6 6 6 6 2 2 2 2 2 2 2 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 3 3 3 3 3 3 3 3 5 5 5 5 5 5 1 1 4 2 2 2 3 3 2 2 4 4 4 4 4 4 4 4 4 4 4 4 4 5 5 5 5 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 6 6 6 6 6 6 7 7 7 7 7 7 7 7 7 7 1 1 3 5 5 5 5 5 5 5 5 5 5 5 5 3 3 3 3 3 4 4 4 4 4 5 5 5 5 1 1 1 1 6 6 6 4 4 4 4 4 4 4 4 5 5 5 7 7 7 7 7 7 7 4 4 4 4 4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 4 4 4 1 1 5 5 5 5 5 5 5 5 1 1 5 5 3 3 4 4 1 1 4 4 5 4 4 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 4 4 4 4 4 6 6 6 6 6 6 6 6 1 1 1 1 1 \n 296 242 242 222 222 222 311 311 311 311 311 311 311 307 307 307 307 307 307 307 307 307 307 307 307 307 307 307 307 307 307 307 307 307 307 279 279 279 279 252 252 233 233 233 233 233 233 233 233 233 243 243 243 243 469 226 313 256 284 284 284 284 284 284 216 337 337 345 345 345 305 305 305 305 398 398 398 398 398 398 281 281 281 281 247 247 247 247 270 270 270 270 270 270 270 276 276 276 276 276 384 384 384 384 384 384 384 384 384 384 384 432 432 432 432 432 432 432 432 432 188 188 188 188 188 188 188 437 437 437 437 437 437 437 437 437 437 437 437 437 437 437 403 403 403 403 403 403 403 403 403 403 403 403 403 403 403 403 403 403 403 403 403 403 403 403 403 403 403 403 403 403 296 296 296 296 296 296 296 193 193 193 193 193 193 193 193 398 398 398 398 398 398 265 265 255 329 329 329 402 402 348 348 224 224 277 277 277 277 277 277 277 277 277 277 277 276 276 276 276 307 307 307 307 307 307 307 307 307 307 307 307 307 307 307 307 307 307 300 300 300 300 300 300 330 330 330 330 330 330 330 330 330 330 315 315 244 264 264 264 264 264 264 264 264 264 264 264 264 223 223 223 223 223 254 254 254 254 254 216 216 216 216 198 285 300 241 293 293 293 245 245 245 289 289 289 289 289 358 358 358 329 329 329 222 222 222 222 304 304 304 304 304 304 304 304 304 304 304 304 287 287 287 287 287 287 287 287 430 430 430 304 304 224 224 224 224 224 224 224 224 284 422 370 370 352 352 351 280 335 335 411 411 187 334 334 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 666 711 711 711 711 711 391 391 391 391 391 391 391 391 273 273 273 273 273 \n 15.3 17.8 17.8 18.7 18.7 18.7 15.2 15.2 15.2 15.2 15.2 15.2 15.2 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 19.2 19.2 19.2 19.2 18.3 18.3 17.9 17.9 17.9 17.9 17.9 17.9 17.9 17.9 17.9 16.8 16.8 16.8 16.8 21.1 17.9 17.3 15.1 19.7 19.7 19.7 19.7 19.7 19.7 18.6 16.1 16.1 18.9 18.9 18.9 19.2 19.2 19.2 19.2 18.7 18.7 18.7 18.7 18.7 18.7 19 19 19 19 18.5 18.5 18.5 18.5 17.8 17.8 17.8 17.8 18.2 18.2 18.2 18 18 18 18 18 20.9 20.9 20.9 20.9 20.9 20.9 20.9 20.9 20.9 20.9 20.9 17.8 17.8 17.8 17.8 17.8 17.8 17.8 17.8 17.8 19.1 19.1 19.1 19.1 19.1 19.1 19.1 21.2 21.2 21.2 21.2 21.2 21.2 21.2 21.2 21.2 21.2 21.2 21.2 21.2 21.2 21.2 14.7 14.7 14.7 14.7 14.7 14.7 14.7 14.7 14.7 14.7 14.7 14.7 14.7 14.7 14.7 14.7 14.7 14.7 14.7 14.7 14.7 14.7 14.7 14.7 14.7 14.7 14.7 14.7 14.7 14.7 16.6 16.6 16.6 16.6 16.6 16.6 16.6 17.8 17.8 17.8 17.8 17.8 17.8 17.8 17.8 15.2 15.2 15.2 15.2 15.2 15.2 15.6 15.6 14.4 12.6 12.6 12.6 17 17 14.7 14.7 14.7 14.7 18.6 18.6 18.6 18.6 18.6 18.6 18.6 18.6 18.6 18.6 18.6 16.4 16.4 16.4 16.4 17.4 17.4 17.4 17.4 17.4 17.4 17.4 17.4 17.4 17.4 17.4 17.4 17.4 17.4 17.4 17.4 17.4 17.4 16.6 16.6 16.6 16.6 16.6 16.6 19.1 19.1 19.1 19.1 19.1 19.1 19.1 19.1 19.1 19.1 16.4 16.4 15.9 13 13 13 13 13 13 13 13 13 13 13 13 18.6 18.6 18.6 18.6 18.6 17.6 17.6 17.6 17.6 17.6 14.9 14.9 14.9 14.9 13.6 15.3 15.3 18.2 16.6 16.6 16.6 19.2 19.2 19.2 16 16 16 16 16 14.8 14.8 14.8 16.1 16.1 16.1 18.4 18.4 18.4 18.4 18.4 18.4 18.4 18.4 18.4 18.4 18.4 18.4 18.4 18.4 18.4 18.4 19.6 19.6 19.6 19.6 19.6 19.6 19.6 19.6 16.9 16.9 16.9 16.9 16.9 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 15.5 15.9 17.6 17.6 18.8 18.8 17.9 17 19.7 19.7 18.3 18.3 17 22 22 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.1 20.1 20.1 20.1 20.1 19.2 19.2 19.2 19.2 19.2 19.2 19.2 19.2 21 21 21 21 21 \n 397 397 393 395 397 394 396 397 387 387 393 397 391 397 380 396 387 387 289 391 377 393 397 395 394 303 377 306 388 380 360 377 233 359 248 397 378 397 393 396 396 385 383 394 389 397 397 393 397 397 396 394 397 397 397 396 397 393 391 397 395 378 397 396 393 397 397 396 397 397 384 377 391 377 395 383 374 387 386 396 397 396 397 391 397 392 396 395 397 396 392 394 395 396 397 358 392 397 394 397 395 396 70.8 394 393 394 396 388 395 391 393 396 395 397 389 345 393 395 339 392 389 378 378 370 379 385 359 392 397 397 395 397 386 389 263 395 378 394 392 397 388 397 397 397 397 173 169 392 357 352 373 342 343 262 321 88 88.6 363 354 364 339 374 390 388 395 240 369 228 297 330 292 348 397 396 393 391 393 396 391 397 396 397 394 397 391 387 393 394 383 397 378 390 390 393 377 394 397 354 392 397 384 394 395 393 391 397 395 389 381 397 393 395 391 386 349 394 393 393 397 394 392 395 390 397 385 382 387 372 378 380 378 376 386 379 360 377 388 390 379 384 391 395 373 375 372 389 390 376 375 394 396 377 386 397 393 395 386 390 383 392 393 388 387 393 388 392 384 385 390 391 389 397 395 391 397 397 389 393 397 397 387 392 377 396 395 395 342 397 397 372 397 397 397 397 397 397 393 397 368 372 391 396 384 390 394 393 397 397 397 396 350 397 396 393 396 396 391 397 395 396 397 397 397 391 397 394 397 397 382 375 369 394 362 390 389 397 397 395 396 397 397 395 390 397 388 386 365 392 391 390 397 371 392 384 383 376 378 391 395 391 375 351 381 353 355 355 316 131 376 375 392 366 348 397 397 397 363 397 397 394 397 397 397 397 286 397 397 397 373 397 394 378 397 397 397 392 397 393 397 338 397 397 376 397 329 385 370 332 315 179 2.6 35.1 28.8 211 88.3 27.3 21.6 127 16.5 48.5 319 320 292 2.52 3.65 7.68 24.7 18.8 96.7 60.7 83.5 81.3 98 100 101 110 27.5 9.32 69 397 391 386 396 387 241 43.1 318 389 397 304 0.32 355 385 376 6.68 50.9 10.5 3.5 272 397 255 391 397 394 397 334 22 331 369 397 397 395 393 375 353 303 396 349 380 383 397 393 395 393 371 389 393 388 395 344 318 390 397 397 397 393 397 397 397 396 397 392 397 397 393 397 \n 4.98 9.14 4.03 2.94 5.33 5.21 12.4 19.2 29.9 17.1 20.5 13.3 15.7 8.26 10.3 8.47 6.58 14.7 11.7 11.3 21 13.8 18.7 19.9 16.3 16.5 14.8 17.3 12.8 12 22.6 13 27.7 18.4 20.3 9.68 11.4 8.77 10.1 4.32 1.98 4.84 5.81 7.44 9.55 10.2 14.2 18.8 30.8 16.2 13.5 9.43 5.28 8.43 14.8 4.81 5.77 3.95 6.86 9.22 13.2 14.4 6.73 9.5 8.05 4.67 10.2 8.1 13.1 8.79 6.72 9.88 5.52 7.54 6.78 8.94 12 10.3 12.3 9.1 5.29 7.22 6.72 7.51 9.62 6.53 12.9 8.44 5.5 5.7 8.81 8.2 8.16 6.21 10.6 6.65 11.3 4.21 3.57 6.19 9.42 7.67 10.6 13.4 12.3 16.5 18.7 14.1 12.3 15.6 13 10.2 16.2 17.1 10.5 15.8 12 10.3 15.4 13.6 14.4 14.3 17.9 25.4 17.6 14.8 27.3 17.2 15.4 18.3 12.6 12.3 11.1 15 17.3 17 16.9 14.6 21.3 18.5 24.2 34.4 26.8 26.4 29.3 27.8 16.7 29.5 28.3 21.5 14.1 13.3 12.1 15.8 15.1 15 16.1 4.59 6.43 7.39 5.5 1.73 1.92 3.32 11.6 9.81 3.7 12.1 11.1 11.3 14.4 12 14.7 9.04 9.64 5.33 10.1 6.29 6.92 5.04 7.56 9.45 4.82 5.68 14 13.2 4.45 6.68 4.56 5.39 5.1 4.69 2.87 5.03 4.38 2.97 4.08 8.61 6.62 4.56 4.45 7.43 3.11 3.81 2.88 10.9 11 18.1 14.7 23.1 17.3 24 16 9.38 29.6 9.47 13.5 9.69 17.9 10.5 9.71 21.5 9.93 7.6 4.14 4.63 3.13 6.36 3.92 3.76 11.7 5.25 2.47 3.95 8.05 10.9 9.54 4.73 6.36 7.37 11.4 12.4 11.2 5.19 12.5 18.5 9.16 10.2 9.52 6.56 5.9 3.59 3.53 3.54 6.57 9.25 3.11 5.12 7.79 6.9 9.59 7.26 5.91 11.3 8.1 10.5 14.8 7.44 3.16 13.7 13 6.59 7.73 6.58 3.53 2.98 6.05 4.16 7.19 4.85 3.76 4.59 3.01 3.16 7.85 8.23 12.9 7.14 7.6 9.51 3.33 3.56 4.7 8.58 10.4 6.27 7.39 15.8 4.97 4.74 6.07 9.5 8.67 4.86 6.93 8.93 6.47 7.53 4.54 9.97 12.6 5.98 11.7 7.9 9.28 11.5 18.3 15.9 10.4 12.7 7.2 6.87 7.7 11.7 6.12 5.08 6.15 12.8 9.97 7.34 9.09 12.4 7.83 5.68 6.75 8.01 9.8 10.6 8.51 9.74 9.29 5.49 8.65 7.18 4.61 10.5 12.7 6.36 5.99 5.89 5.98 5.49 7.79 4.5 8.05 5.57 17.6 13.3 11.5 12.7 7.79 14.2 10.2 14.6 5.29 7.12 14 13.3 3.26 3.73 2.96 9.53 8.88 34.8 38 13.4 23.2 21.2 23.7 21.8 17.2 21.1 23.6 24.6 30.6 30.8 28.3 32 30.6 20.9 17.1 18.8 25.7 15.2 16.4 17.1 19.4 19.9 30.6 30 26.8 20.3 20.3 19.8 27.4 23 23.3 12.1 26.4 19.8 10.1 21.2 34.4 20.1 37 29.1 25.8 26.6 20.6 22.7 15 15.7 14.1 23.3 17.2 24.4 15.7 14.5 21.5 24.1 17.6 19.7 12 16.2 15.2 23.3 18.1 26.5 34 22.9 22.1 19.5 16.6 18.9 23.8 24 17.8 16.4 18.1 19.3 17.4 17.7 17.3 16.7 18.7 18.1 19 16.9 16.2 14.7 16.4 14.7 14 10.3 13.2 14.1 17.2 21.3 18.1 14.8 16.3 12.9 14.4 11.7 18.1 24.1 18.7 24.9 18 13.1 10.7 7.74 7.01 10.4 13.3 10.6 15 11.5 18.1 24 29.7 18.1 13.4 12 13.6 17.6 21.1 14.1 12.9 15.1 14.3 9.67 9.08 5.64 6.48 7.88 \n 24 21.6 34.7 33.4 36.2 28.7 22.9 27.1 16.5 18.9 15 18.9 21.7 20.4 18.2 19.9 23.1 17.5 20.2 18.2 13.6 19.6 15.2 14.5 15.6 13.9 16.6 14.8 18.4 21 12.7 14.5 13.2 13.1 13.5 18.9 20 21 24.7 30.8 34.9 26.6 25.3 24.7 21.2 19.3 20 16.6 14.4 19.4 19.7 20.5 25 23.4 18.9 35.4 24.7 31.6 23.3 19.6 18.7 16 22.2 25 33 23.5 19.4 22 17.4 20.9 24.2 21.7 22.8 23.4 24.1 21.4 20 20.8 21.2 20.3 28 23.9 24.8 22.9 23.9 26.6 22.5 22.2 23.6 28.7 22.6 22 22.9 25 20.6 28.4 21.4 38.7 43.8 33.2 27.5 26.5 18.6 19.3 20.1 19.5 19.5 20.4 19.8 19.4 21.7 22.8 18.8 18.7 18.5 18.3 21.2 19.2 20.4 19.3 22 20.3 20.5 17.3 18.8 21.4 15.7 16.2 18 14.3 19.2 19.6 23 18.4 15.6 18.1 17.4 17.1 13.3 17.8 14 14.4 13.4 15.6 11.8 13.8 15.6 14.6 17.8 15.4 21.5 19.6 15.3 19.4 17 15.6 13.1 41.3 24.3 23.3 27 50 50 50 22.7 25 50 23.8 23.8 22.3 17.4 19.1 23.1 23.6 22.6 29.4 23.2 24.6 29.9 37.2 39.8 36.2 37.9 32.5 26.4 29.6 50 32 29.8 34.9 37 30.5 36.4 31.1 29.1 50 33.3 30.3 34.6 34.9 32.9 24.1 42.3 48.5 50 22.6 24.4 22.5 24.4 20 21.7 19.3 22.4 28.1 23.7 25 23.3 28.7 21.5 23 26.7 21.7 27.5 30.1 44.8 50 37.6 31.6 46.7 31.5 24.3 31.7 41.7 48.3 29 24 25.1 31.5 23.7 23.3 22 20.1 22.2 23.7 17.6 18.5 24.3 20.5 24.5 26.2 24.4 24.8 29.6 42.8 21.9 20.9 44 50 36 30.1 33.8 43.1 48.8 31 36.5 22.8 30.7 50 43.5 20.7 21.1 25.2 24.4 35.2 32.4 32 33.2 33.1 29.1 35.1 45.4 35.4 46 50 32.2 22 20.1 23.2 22.3 24.8 28.5 37.3 27.9 23.9 21.7 28.6 27.1 20.3 22.5 29 24.8 22 26.4 33.1 36.1 28.4 33.4 28.2 22.8 20.3 16.1 22.1 19.4 21.6 23.8 16.2 17.8 19.8 23.1 21 23.8 23.1 20.4 18.5 25 24.6 23 22.2 19.3 22.6 19.8 17.1 19.4 22.2 20.7 21.1 19.5 18.5 20.6 19 18.7 32.7 16.5 23.9 31.2 17.5 17.2 23.1 24.5 26.6 22.9 24.1 18.6 30.1 18.2 20.6 17.8 21.7 22.7 22.6 25 19.9 20.8 16.8 21.9 27.5 21.9 23.1 50 50 50 50 50 13.8 13.8 15 13.9 13.3 13.1 10.2 10.4 10.9 11.3 12.3 8.8 7.2 10.5 7.4 10.2 11.5 15.1 23.2 9.7 13.8 12.7 13.1 12.5 8.5 5 6.3 5.6 7.2 12.1 8.3 8.5 5 11.9 27.9 17.2 27.5 15 17.2 17.9 16.3 7 7.2 7.5 10.4 8.8 8.4 16.7 14.2 20.8 13.4 11.7 8.3 10.2 10.9 11 9.5 14.5 14.1 16.1 14.3 11.7 13.4 9.6 8.7 8.4 12.8 10.5 17.1 18.4 15.4 10.8 11.8 14.9 12.6 14.1 13 13.4 15.2 16.1 17.8 14.9 14.1 12.7 13.5 14.9 20 16.4 17.7 19.5 20.2 21.4 19.9 19 19.1 19.1 20.1 19.9 19.6 23.2 29.8 13.8 13.3 16.7 12 14.6 21.4 23 23.7 25 21.8 20.6 21.2 19.1 20.6 15.2 7 8.1 13.6 20.1 21.8 24.5 23.1 19.7 18.3 21.2 17.5 16.8 22.4 20.6 23.9 22 11.9 \n\n\nThe data has 14 rows and 506 columns, where each column represents the 14 features of one house. The values in the last row represent the price of the property, and we will train a model to predict this value, given the remaining 13 features.\n\nWe also add a row of ones to the input matrix that will account for the bias (intercept) of our model and simplify the implementation.\n\n*)\n\nlet hx = h.[0..12, *]\nlet hy = h.[13..13, *]\n\nlet housing = Dataset(hx, hy).AppendBiasRowX()\n\n(**\n\nOur linear regression model is of the form\n\n$$$\n h_{\\mathbf{w}} (\\mathbf{x}) = \\sum_j w_j x_j = \\mathbf{w}^{T} \\mathbf{x}\n\nwhich represents a family of linear functions parameterized by the vector $\\mathbf{w}$. \n*)\n\nlet model (w:DV) (x:DV) = w * x\n\n(**\n\nFor training the model, we minimize a loss function\n\n$$$\n J(\\mathbf{w}) = \\frac{1}{2} \\sum_{i=1}^{n} \\left(h_{\\mathbf{w}} (\\mathbf{x}^{(i)}) - y^{(i)} \\right)^2 = \\frac{1}{2} \\sum_{i=1}^{n} \\left( \\mathbf{w}^{T} \\mathbf{x}^{(i)} - y^{(i)} \\right)^2\n\nwhere $\\mathbf{x}^{(i)}$ are vectors holding the 13 input features plus the bias input (the constant 1) and $y^{(i)}$ are the target values (which are here scalar).\n\n*)\n\nlet wopt, lopt, whist, lhist = Optimize.Train(model, Rnd.UniformDV(14), housing, \n {Params.Default with Epochs = 1000\n Loss = Loss.Quadratic})\n\nlet trainedmodel = model wopt\n\n\n(*** hide ***)\nopen RProvider\nopen RProvider.graphics\nopen RProvider.grDevices\n\nlet px, py = housing.Y.[0,*] |> DV.toArray |> Array.mapi (fun i v -> float i, (v |> float32 |> float)) |> Array.unzip\nlet ppx, ppy = housing.X |> DM.mapCols (fun v -> toDV [trainedmodel v]) |> DM.toDV |> DV.toArray |> Array.mapi (fun i v -> float i, (v |> float32 |> float)) |> Array.unzip\n\nlet ll = lhist |> Array.map (float32>>float)\n\nnamedParams[\n \"x\", box px\n \"y\", box py\n \"pch\", box 19\n \"col\", box \"darkblue\"\n \"type\", box \"p\"\n \"xlab\", box \"House number\"\n \"ylab\", box \"Price\"\n \"width\", box 700\n \"height\", box 500\n ]\n|> R.plot|> ignore\n\nnamedParams[\n \"x\", box ppx\n \"y\", box ppy\n \"pch\", box 19\n \"col\", box \"red\"\n \"type\", box \"p\"\n \"width\", box 700\n \"height\", box 500\n ]\n|> R.points|> ignore\n\n(**\n\n
\n[12/11/2015 14:41:04] --- Training started\n[12/11/2015 14:41:04] Parameters : 14\n[12/11/2015 14:41:04] Iterations : 1000\n[12/11/2015 14:41:04] Epochs : 1000\n[12/11/2015 14:41:04] Batches : Full (1 per epoch)\n[12/11/2015 14:41:04] Training data : 506\n[12/11/2015 14:41:04] Validation data: None\n[12/11/2015 14:41:04] Valid. interval: 10\n[12/11/2015 14:41:04] Method : Gradient descent\n[12/11/2015 14:41:04] Learning rate : RMSProp a0 = D 0.00100000005f, k = D 0.899999976f\n[12/11/2015 14:41:04] Momentum : None\n[12/11/2015 14:41:04] Loss : L2 norm\n[12/11/2015 14:41:04] Regularizer : L2 lambda = D 9.99999975e-05f\n[12/11/2015 14:41:04] Gradient clip. : None\n[12/11/2015 14:41:04] Early stopping : None\n[12/11/2015 14:41:04] Improv. thresh.: D 0.995000005f\n[12/11/2015 14:41:04] Return best : true\n[12/11/2015 14:41:04] 1/1000 | Batch 1/1 | D 5.281104e+002 [- ]\n[12/11/2015 14:41:04] 2/1000 | Batch 1/1 | D 5.252324e+002 [↓▼]\n[12/11/2015 14:41:04] 3/1000 | Batch 1/1 | D 5.231447e+002 [↓ ]\n[12/11/2015 14:41:04] 4/1000 | Batch 1/1 | D 5.213967e+002 [↓▼]\n[12/11/2015 14:41:04] 5/1000 | Batch 1/1 | D 5.198447e+002 [↓ ]\n[12/11/2015 14:41:04] 6/1000 | Batch 1/1 | D 5.184225e+002 [↓▼]\n[12/11/2015 14:41:04] 7/1000 | Batch 1/1 | D 5.170928e+002 [↓ ]\n...\n[12/11/2015 14:41:27] 994/1000 | Batch 1/1 | D 6.404338e+000 [↓ ]\n[12/11/2015 14:41:27] 995/1000 | Batch 1/1 | D 6.392090e+000 [↓ ]\n[12/11/2015 14:41:27] 996/1000 | Batch 1/1 | D 6.377205e+000 [↓▼]\n[12/11/2015 14:41:28] 997/1000 | Batch 1/1 | D 6.363370e+000 [↓ ]\n[12/11/2015 14:41:28] 998/1000 | Batch 1/1 | D 6.351198e+000 [↓ ]\n[12/11/2015 14:41:28] 999/1000 | Batch 1/1 | D 6.344284e+000 [↓▼]\n[12/11/2015 14:41:28] 1000/1000 | Batch 1/1 | D 6.334455e+000 [↓ ]\n[12/11/2015 14:41:28] Duration : 00:00:23.3076639\n[12/11/2015 14:41:28] Loss initial : D 5.281104e+002\n[12/11/2015 14:41:28] Loss final : D 6.344284e+000 (Best)\n[12/11/2015 14:41:28] Loss change : D -5.217661e+002 (-98.80 %)\n[12/11/2015 14:41:28] Loss chg. / s : D -2.238603e+001\n[12/11/2015 14:41:28] Epochs / s : 42.90434272\n[12/11/2015 14:41:28] Epochs / min : 2574.260563\n[12/11/2015 14:41:28] --- Training finished\n\nval trainedmodel : (DV -> D)\n\n\nThe following is a plot of the prices in the dataset where the blue points represent the real price, and the red points are the values predicted by the trained linear model.\n\n
![\"Chart\"](\"img/Training-1.png\")
\n \nval housingtrain : Dataset = Hype.Dataset\n X: 14 x 400\n Y: 1 x 400\nval housingvalid : Dataset = Hype.Dataset\n X: 14 x 106\n Y: 1 x 106\n\n*)\n\nlet wopt, lopt, whist, lhist = Optimize.Train(model, Rnd.UniformDV(14), housingtrain, housingvalid,\n {Params.Default with \n Epochs = 1000;\n EarlyStopping = Hype.EarlyStopping.Early(750, 10)})\n\n(**\n
\n[12/11/2015 15:09:15] --- Training started\n[12/11/2015 15:09:15] Parameters : 14\n[12/11/2015 15:09:15] Iterations : 1000\n[12/11/2015 15:09:15] Epochs : 1000\n[12/11/2015 15:09:15] Batches : Full (1 per epoch)\n[12/11/2015 15:09:15] Training data : 400\n[12/11/2015 15:09:15] Validation data: 106\n[12/11/2015 15:09:15] Valid. interval: 10\n[12/11/2015 15:09:15] Method : Gradient descent\n[12/11/2015 15:09:15] Learning rate : RMSProp a0 = D 0.00100000005f, k = D 0.899999976f\n[12/11/2015 15:09:15] Momentum : None\n[12/11/2015 15:09:15] Loss : L2 norm\n[12/11/2015 15:09:15] Regularizer : L2 lambda = D 9.99999975e-05f\n[12/11/2015 15:09:15] Gradient clip. : None\n[12/11/2015 15:09:15] Early stopping : Stagnation thresh. = 750, overfit. thresh. = 10\n[12/11/2015 15:09:15] Improv. thresh.: D 0.995000005f\n[12/11/2015 15:09:15] Return best : true\n[12/11/2015 15:09:15] 1/1000 | Batch 1/1 | D 3.221269e+002 [- ] | Valid D 3.322605e+002 [- ] | Stag: 0 Ovfit: 0\n[12/11/2015 15:09:15] 2/1000 | Batch 1/1 | D 3.193867e+002 [↓▼] | Valid D 3.288632e+002 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 15:09:15] 3/1000 | Batch 1/1 | D 3.173987e+002 [↓▼] | Valid D 3.263986e+002 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 15:09:15] 4/1000 | Batch 1/1 | D 3.157341e+002 [↓▼] | Valid D 3.243348e+002 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 15:09:15] 5/1000 | Batch 1/1 | D 3.142565e+002 [↓ ] | Valid D 3.225029e+002 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 15:09:15] 6/1000 | Batch 1/1 | D 3.129025e+002 [↓▼] | Valid D 3.208241e+002 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 15:09:15] 7/1000 | Batch 1/1 | D 3.116365e+002 [↓ ] | Valid D 3.192545e+002 [↓ ] | Stag: 10 Ovfit: 0\n[12/11/2015 15:09:15] 8/1000 | Batch 1/1 | D 3.104370e+002 [↓▼] | Valid D 3.177671e+002 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 15:09:15] 9/1000 | Batch 1/1 | D 3.092885e+002 [↓ ] | Valid D 3.163436e+002 [↓ ] | Stag: 10 Ovfit: 0\n[12/11/2015 15:09:15] 10/1000 | Batch 1/1 | D 3.081814e+002 [↓▼] | Valid D 3.149709e+002 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 15:09:15] 11/1000 | Batch 1/1 | D 3.071076e+002 [↓ ] | Valid D 3.136398e+002 [↓ ] | Stag: 10 Ovfit: 0\n[12/11/2015 15:09:15] 12/1000 | Batch 1/1 | D 3.060618e+002 [↓▼] | Valid D 3.123428e+002 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 15:09:15] 13/1000 | Batch 1/1 | D 3.050388e+002 [↓ ] | Valid D 3.110746e+002 [↓ ] | Stag: 10 Ovfit: 0\n...\n[12/11/2015 15:09:21] 318/1000 | Batch 1/1 | D 4.250416e+001 [↓▼] | Valid D 3.382476e+001 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 15:09:21] 319/1000 | Batch 1/1 | D 4.178834e+001 [↓▼] | Valid D 3.371201e+001 [↓ ] | Stag: 10 Ovfit: 0\n[12/11/2015 15:09:21] 320/1000 | Batch 1/1 | D 4.109373e+001 [↓▼] | Valid D 3.361367e+001 [↓▼] | Stag: 0 Ovfit: 0\n[12/11/2015 15:09:21] 321/1000 | Batch 1/1 | D 4.040976e+001 [↓▼] | Valid D 3.362166e+001 [↑ ] | Stag: 10 Ovfit: 0\n[12/11/2015 15:09:21] 322/1000 | Batch 1/1 | D 3.973472e+001 [↓▼] | Valid D 3.368684e+001 [↑ ] | Stag: 20 Ovfit: 1\n[12/11/2015 15:09:21] 323/1000 | Batch 1/1 | D 3.907929e+001 [↓▼] | Valid D 3.382304e+001 [↑ ] | Stag: 30 Ovfit: 2\n[12/11/2015 15:09:21] 324/1000 | Batch 1/1 | D 3.845267e+001 [↓▼] | Valid D 3.398524e+001 [↑ ] | Stag: 40 Ovfit: 3\n[12/11/2015 15:09:21] 325/1000 | Batch 1/1 | D 3.783842e+001 [↓▼] | Valid D 3.418199e+001 [↑ ] | Stag: 50 Ovfit: 4\n[12/11/2015 15:09:21] 326/1000 | Batch 1/1 | D 3.721857e+001 [↓▼] | Valid D 3.450164e+001 [↑ ] | Stag: 60 Ovfit: 5\n[12/11/2015 15:09:21] 327/1000 | Batch 1/1 | D 3.659464e+001 [↓▼] | Valid D 3.499456e+001 [↑ ] | Stag: 70 Ovfit: 6\n[12/11/2015 15:09:21] 328/1000 | Batch 1/1 | D 3.598552e+001 [↓▼] | Valid D 3.556280e+001 [↑ ] | Stag: 80 Ovfit: 7\n[12/11/2015 15:09:21] 329/1000 | Batch 1/1 | D 3.538885e+001 [↓▼] | Valid D 3.616002e+001 [↑ ] | Stag: 90 Ovfit: 8\n[12/11/2015 15:09:21] 330/1000 | Batch 1/1 | D 3.481464e+001 [↓▼] | Valid D 3.678414e+001 [↑ ] | Stag:100 Ovfit: 9\n[12/11/2015 15:09:21] *** EARLY STOPPING TRIGGERED: Overfitting ***\n[12/11/2015 15:09:21] 331/1000 | Batch 1/1 | D 3.426452e+001 [↓▼] | Valid D 3.741238e+001 [↑ ] | Stag:110 Ovfit:10\n[12/11/2015 15:09:21] Duration : 00:00:05.9617220\n[12/11/2015 15:09:21] Loss initial : D 3.221269e+002\n[12/11/2015 15:09:21] Loss final : D 3.373809e+001 (Best)\n[12/11/2015 15:09:21] Loss change : D -2.883888e+002 (-89.53 %)\n[12/11/2015 15:09:21] Loss chg. / s : D -4.837340e+001\n[12/11/2015 15:09:21] Epochs / s : 55.52087132\n[12/11/2015 15:09:21] Epochs / min : 3331.252279\n[12/11/2015 15:09:21] --- Training finished\n\n*)\n" }, { "path": "docs/input/download.fsx", "content": "(**\nDownload\n========\n\nHype is tested on Linux and Windows.\n\nYou can download the source code or the binaries of the [latest release on GitHub](https://github.com/hypelib/Hype/releases).\n\nYou can also install the library as a package through [NuGet](https://www.nuget.org/packages/Hype), by running
Install-Package Hypein the package manager console.\n\nHype only supports the 64 bit platform, so please make sure that you set \"x64\" as the platform target. For detailed instructions, please see the installation instructions of [DiffSharp](http://diffsharp.github.io/DiffSharp/), on which Hype depends.\n*)" }, { "path": "docs/input/files/misc/style.css", "content": "@import url(https://fonts.googleapis.com/css?family=Droid+Sans|Droid+Sans+Mono|Open+Sans:400,600,700);\n\n/*-------------------------------------------------------------------------- \n Formatting for F# code snippets \n/*--------------------------------------------------------------------------*/\n\n/* strings --- and stlyes for other string related formats */\nspan.s { color:#E0E268; }\n/* printf formatters */\nspan.pf { color:#E0C57F; }\n/* escaped chars */\nspan.e { color:#EA8675; }\n\n/* identifiers --- and styles for more specific identifier types */\nspan.i { color:#d1d1d1; }\n/* type or module */\nspan.t { color:#43AEC6; }\n/* function */\nspan.f { color:#e1e1e1; }\n/* DU case or active pattern */\nspan.p { color:#4ec9b0; }\n\n/* keywords */\nspan.k { color:#FAB11D; }\n/* comment */\nspan.c { color:#808080; }\n/* operators */\nspan.o { color:#af75c1; }\n/* numbers */\nspan.n { color:#96C71D; }\n/* line number */\nspan.l { color:#80b0b0; }\n/* mutable var or ref cell */\nspan.v { color:#d1d1d1; font-weight: bold; }\n/* inactive code */\nspan.inactive { color:#808080; }\n/* preprocessor */\nspan.prep { color:#af75c1; }\n/* fsi output */\nspan.fsi { color:#808080; }\n\n/* omitted */\nspan.omitted { \n\tbackground:#3c4e52;\n border-radius:5px;\n\tcolor:#808080;\n\tpadding:0px 0px 1px 0px;\n}\n/* tool tip */\ndiv.tip {\n\tbackground:#475b5f;\n border-radius:4px;\n font:11pt 'Droid Sans', arial, sans-serif;\n\tpadding:6px 8px 6px 8px;\n\tdisplay:none;\n color:#d1d1d1;\n}\ntable.pre pre {\n padding:0px;\n margin:0px;\n border:none;\n} \ntable.pre, pre.fssnip, pre {\n line-height:13pt;\n border:1px solid #191919;\n border-collapse:separate;\n white-space:pre;\n font: 9pt 'Droid Sans Mono',consolas,monospace;\n width:90%;\n margin:10px 20px 20px 20px;\n background-color:#212d30;\n padding:10px;\n border-radius:5px;\n color:#d1d1d1; \n}\ntable.pre pre {\n padding:0px;\n margin:0px;\n border-radius:0px;\n width: 100%;\n}\ntable.pre td {\n padding:0px;\n white-space:normal;\n margin:0px;\n}\ntable.pre td.lines {\n width:30px;\n}\n\n/*-------------------------------------------------------------------------- \n Formatting for page & standard document content\n/*--------------------------------------------------------------------------*/\n\nbody {\n font-family: 'Open Sans', 'Segoe UI', serif;\n padding-top: 0px;\n padding-bottom: 40px;\n color: #dbdbdb;\n background-color: #191919;\n}\n\npre {\n word-wrap: inherit;\n}\n\n.nav-pills > li > a {\n background-color: #191919;\n font-size: 11pt;\n height: 32px;\n padding: 7px 15px;\n}\n\n.nav-pills > li > a:hover {\n color : #000000;\n background-color: #3781c3;\n font-size: 11pt;\n}\n\n.alert-info {\n color: #ebebeb;\n background-color: #1f1f1f;\n border-color: #191919;\n}\n\n.hype {\n color: #fff;\n}\n\n.nav-label {\n font-size: 11pt;\n height: 34px;\n padding: 7px 15px;\n margin: 20px 0px 0px 0px;\n color: #dbdbdb;\n}\n\na:link {\n color: #3781c3;\n}\n\na:visited {\n color: #3275b0;\n}\n\na:hover {\n color: #7fb0db;\n}\n\na:active {\n color: #35ad67;\n}\n\nh1 {\n margin-bottom: 30px;\n}\n\n/* Make table headings and td.title bold */\ntd.title, thead {\n font-weight:bold;\n}\n\n\n/*-------------------------------------------------------------------------- \n Formatting for API reference\n/*--------------------------------------------------------------------------*/\n\n.type-list .type-name, .module-list .module-name {\n width:25%;\n font-weight:bold;\n}\n.member-list .member-name {\n width:35%;\n}\n#main .xmldoc h2 {\n font-size:14pt;\n margin:10px 0px 0px 0px;\n}\n#main .xmldoc h3 {\n font-size:12pt;\n margin:10px 0px 0px 0px;\n}\n.github-link {\n float:right;\n text-decoration:none;\n}\n.github-link img {\n border-style:none;\n margin-left:10px;\n}\n.github-link .hover { display:none; }\n.github-link:hover .hover { display:block; }\n.github-link .normal { display: block; }\n.github-link:hover .normal { display: none; }\n\ncode {\n color: white;\n background-color: #2f2f2f;\n font-size: 100%;\n}\n\n/*-------------------------------------------------------------------------- \n Links \n/*--------------------------------------------------------------------------*/\n\nh1 a, h1 a:hover, h1 a:focus,\nh2 a, h2 a:hover, h2 a:focus,\nh3 a, h3 a:hover, h3 a:focus,\nh4 a, h4 a:hover, h4 a:focus,\nh5 a, h5 a:hover, h5 a:focus,\nh6 a, h6 a:hover, h6 a:focus { color : inherit; text-decoration : inherit; outline:none }\n\n/*-------------------------------------------------------------------------- \n Additional formatting for the homepage \n/*--------------------------------------------------------------------------*/\n\n#nuget { \n margin-top:20px;\n font-size: 11pt; \n padding:20px;\n background: #e0e0e0;\n color: #3c3c3c;\n}\n\n#nuget pre {\n font-size:11pt;\n -moz-border-radius: 0px;\n -webkit-border-radius: 0px;\n border-radius: 0px;\n background: #3c3c3c;\n border-style:none;\n color: #e0e0e0;\n margin-top:15px;\n}" }, { "path": "docs/input/files/misc/style_light.css", "content": "@import url(https://fonts.googleapis.com/css?family=Droid+Sans|Droid+Sans+Mono|Open+Sans:400,600,700);\n\n/*-------------------------------------------------------------------------- \n Formatting for F# code snippets \n/*--------------------------------------------------------------------------*/\n\n/* identifier */\nspan.i { color:#000000; }\n/* string */\nspan.s { color:#a31515; }\n/* keywords */\nspan.k { color:#0000ff; }\n/* comment */\nspan.c { color:#008000; }\n/* operators */\nspan.o { color:#000000; }\n/* numbers */\nspan.n { color:#000000; }\n/* line number */\nspan.l { color:#96c2cd; }\n/* type or module */\nspan.t { color:#2b91af; }\n/* function */\nspan.f { color:#0000a0; }\n/* DU case or active pattern */\nspan.p { color:#800080; }\n/* mutable var or ref cell */\nspan.v { color:#000000; font-weight: bold; }\n/* printf formatters */\nspan.pf { color:#2b91af; }\n/* escaped chars */\nspan.e { color:#ff0080; }\n/* mutable var or ref cell */\n\n\n/* inactive code */\nspan.inactive { color:#808080; }\n/* preprocessor */\nspan.prep { color:#0000ff; }\n/* fsi output */\nspan.fsi { color:#808080; }\n\n/* omitted */\nspan.omitted { \n\tbackground:#3c4e52;\n border-radius:5px;\n\tcolor:#808080;\n\tpadding:0px 0px 1px 0px;\n}\n/* tool tip */\ndiv.tip {\n\tbackground:#e5e5e5;\n border-radius:4px;\n font:9pt 'Droid Sans', arial, sans-serif;\n\tpadding:6px 8px 6px 8px;\n\tdisplay:none;\n color:#000000;\n}\ntable.pre pre {\n padding:0px;\n margin:0px;\n border:none;\n} \ntable.pre, pre.fssnip, pre {\n line-height:13pt;\n border:1px solid #d8d8d8;\n border-collapse:separate;\n white-space:pre;\n font: 10pt consolas,monospace;\n width:90%;\n margin:10px 20px 20px 20px;\n background-color:#fdfdfd;\n padding:10px;\n border-radius:5px;\n color:#000000; \n}\ntable.pre pre {\n padding:0px;\n margin:0px;\n border-radius:0px;\n width: 100%;\n}\ntable.pre td {\n padding:0px;\n white-space:normal;\n margin:0px;\n}\ntable.pre td.lines {\n width:30px;\n}\n\n/*-------------------------------------------------------------------------- \n Formatting for page & standard document content\n/*--------------------------------------------------------------------------*/\n\nbody {\n font-family: 'Open Sans', serif;\n padding-top: 0px;\n padding-bottom: 40px;\n}\n\npre {\n word-wrap: inherit;\n}\n\n/* Format the heading - nicer spacing etc. */\n.masthead {\n overflow: hidden;\n}\n.masthead .muted a {\n text-decoration:none;\n color:#999999;\n}\n.masthead ul, .masthead li {\n margin-bottom:0px;\n}\n.masthead .nav li {\n margin-top: 15px;\n font-size:110%;\n}\n.masthead h3 {\n margin-bottom:5px;\n font-size:170%;\n}\nhr {\n margin:0px 0px 20px 0px;\n}\n\n/* Make table headings and td.title bold */\ntd.title, thead {\n font-weight:bold;\n}\n\n/* Format the right-side menu */\n#menu {\n margin-top:50px;\n font-size:11pt;\n padding-left:20px;\n}\n\n#menu .nav-header {\n font-size:12pt;\n color:#606060;\n margin-top:20px;\n}\n\n#menu li {\n line-height:25px;\n}\n\n/* Change font sizes for headings etc. */\n#main h1 { font-size: 26pt; margin:10px 0px 15px 0px; font-weight:400; }\n#main h2 { font-size: 20pt; margin:20px 0px 0px 0px; font-weight:400; }\n#main h3 { font-size: 14pt; margin:15px 0px 0px 0px; font-weight:600; }\n#main p { font-size: 11pt; margin:5px 0px 15px 0px; }\n#main ul { font-size: 11pt; margin-top:10px; }\n#main li { font-size: 11pt; margin: 5px 0px 5px 0px; }\n#main strong { font-weight:700; }\n\n/*-------------------------------------------------------------------------- \n Formatting for API reference\n/*--------------------------------------------------------------------------*/\n\n.type-list .type-name, .module-list .module-name {\n width:25%;\n font-weight:bold;\n}\n.member-list .member-name {\n width:35%;\n}\n#main .xmldoc h2 {\n font-size:14pt;\n margin:10px 0px 0px 0px;\n}\n#main .xmldoc h3 {\n font-size:12pt;\n margin:10px 0px 0px 0px;\n}\n.github-link {\n float:right;\n text-decoration:none;\n}\n.github-link img {\n border-style:none;\n margin-left:10px;\n}\n.github-link .hover { display:none; }\n.github-link:hover .hover { display:block; }\n.github-link .normal { display: block; }\n.github-link:hover .normal { display: none; }\n\n/*-------------------------------------------------------------------------- \n Links \n/*--------------------------------------------------------------------------*/\n\nh1 a, h1 a:hover, h1 a:focus,\nh2 a, h2 a:hover, h2 a:focus,\nh3 a, h3 a:hover, h3 a:focus,\nh4 a, h4 a:hover, h4 a:focus,\nh5 a, h5 a:hover, h5 a:focus,\nh6 a, h6 a:hover, h6 a:focus { color : inherit; text-decoration : inherit; outline:none }\n\n/*-------------------------------------------------------------------------- \n Additional formatting for the homepage \n/*--------------------------------------------------------------------------*/\n\n#nuget { \n margin-top:20px;\n font-size: 11pt; \n padding:20px; \n}\n\n#nuget pre {\n font-size:11pt;\n -moz-border-radius: 0px;\n -webkit-border-radius: 0px;\n border-radius: 0px;\n background: #404040;\n border-style:none;\n color: #e0e0e0;\n margin-top:15px;\n}" }, { "path": "docs/input/files/misc/tips.js", "content": "var currentTip = null;\nvar currentTipElement = null;\n\nfunction hideTip(evt, name, unique) {\n var el = document.getElementById(name);\n el.style.display = \"none\";\n currentTip = null;\n}\n\nfunction findPos(obj) {\n // no idea why, but it behaves differently in webbrowser component\n if (window.location.search == \"?inapp\")\n return [obj.offsetLeft + 10, obj.offsetTop + 30];\n\n var curleft = 0;\n var curtop = obj.offsetHeight;\n while (obj) {\n curleft += obj.offsetLeft;\n curtop += obj.offsetTop;\n obj = obj.offsetParent;\n };\n return [curleft, curtop];\n}\n\nfunction hideUsingEsc(e) {\n if (!e) { e = event; }\n hideTip(e, currentTipElement, currentTip);\n}\n\nfunction showTip(evt, name, unique, owner) {\n document.onkeydown = hideUsingEsc;\n if (currentTip == unique) return;\n currentTip = unique;\n currentTipElement = name;\n\n var pos = findPos(owner ? owner : (evt.srcElement ? evt.srcElement : evt.target));\n var posx = pos[0];\n var posy = pos[1];\n\n var el = document.getElementById(name);\n var parent = (document.documentElement == null) ? document.body : document.documentElement;\n el.style.position = \"absolute\";\n el.style.left = posx + \"px\";\n el.style.top = posy + \"px\";\n el.style.display = \"block\";\n}" }, { "path": "docs/input/housing.data", "content": "0.00632\t18.00\t2.310\t0\t0.5380\t6.5750\t65.20\t4.0900\t1\t296.0\t15.30\t396.90\t4.98\t24.00\n0.02731\t0.00\t7.070\t0\t0.4690\t6.4210\t78.90\t4.9671\t2\t242.0\t17.80\t396.90\t9.14\t21.60\n0.02729\t0.00\t7.070\t0\t0.4690\t7.1850\t61.10\t4.9671\t2\t242.0\t17.80\t392.83\t4.03\t34.70\n0.03237\t0.00\t2.180\t0\t0.4580\t6.9980\t45.80\t6.0622\t3\t222.0\t18.70\t394.63\t2.94\t33.40\n0.06905\t0.00\t2.180\t0\t0.4580\t7.1470\t54.20\t6.0622\t3\t222.0\t18.70\t396.90\t5.33\t36.20\n0.02985\t0.00\t2.180\t0\t0.4580\t6.4300\t58.70\t6.0622\t3\t222.0\t18.70\t394.12\t5.21\t28.70\n0.08829\t12.50\t7.870\t0\t0.5240\t6.0120\t66.60\t5.5605\t5\t311.0\t15.20\t395.60\t12.43\t22.90\n0.14455\t12.50\t7.870\t0\t0.5240\t6.1720\t96.10\t5.9505\t5\t311.0\t15.20\t396.90\t19.15\t27.10\n0.21124\t12.50\t7.870\t0\t0.5240\t5.6310\t100.00\t6.0821\t5\t311.0\t15.20\t386.63\t29.93\t16.50\n0.17004\t12.50\t7.870\t0\t0.5240\t6.0040\t85.90\t6.5921\t5\t311.0\t15.20\t386.71\t17.10\t18.90\n0.22489\t12.50\t7.870\t0\t0.5240\t6.3770\t94.30\t6.3467\t5\t311.0\t15.20\t392.52\t20.45\t15.00\n0.11747\t12.50\t7.870\t0\t0.5240\t6.0090\t82.90\t6.2267\t5\t311.0\t15.20\t396.90\t13.27\t18.90\n0.09378\t12.50\t7.870\t0\t0.5240\t5.8890\t39.00\t5.4509\t5\t311.0\t15.20\t390.50\t15.71\t21.70\n0.62976\t0.00\t8.140\t0\t0.5380\t5.9490\t61.80\t4.7075\t4\t307.0\t21.00\t396.90\t8.26\t20.40\n0.63796\t0.00\t8.140\t0\t0.5380\t6.0960\t84.50\t4.4619\t4\t307.0\t21.00\t380.02\t10.26\t18.20\n0.62739\t0.00\t8.140\t0\t0.5380\t5.8340\t56.50\t4.4986\t4\t307.0\t21.00\t395.62\t8.47\t19.90\n1.05393\t0.00\t8.140\t0\t0.5380\t5.9350\t29.30\t4.4986\t4\t307.0\t21.00\t386.85\t6.58\t23.10\n0.78420\t0.00\t8.140\t0\t0.5380\t5.9900\t81.70\t4.2579\t4\t307.0\t21.00\t386.75\t14.67\t17.50\n0.80271\t0.00\t8.140\t0\t0.5380\t5.4560\t36.60\t3.7965\t4\t307.0\t21.00\t288.99\t11.69\t20.20\n0.72580\t0.00\t8.140\t0\t0.5380\t5.7270\t69.50\t3.7965\t4\t307.0\t21.00\t390.95\t11.28\t18.20\n1.25179\t0.00\t8.140\t0\t0.5380\t5.5700\t98.10\t3.7979\t4\t307.0\t21.00\t376.57\t21.02\t13.60\n0.85204\t0.00\t8.140\t0\t0.5380\t5.9650\t89.20\t4.0123\t4\t307.0\t21.00\t392.53\t13.83\t19.60\n1.23247\t0.00\t8.140\t0\t0.5380\t6.1420\t91.70\t3.9769\t4\t307.0\t21.00\t396.90\t18.72\t15.20\n0.98843\t0.00\t8.140\t0\t0.5380\t5.8130\t100.00\t4.0952\t4\t307.0\t21.00\t394.54\t19.88\t14.50\n0.75026\t0.00\t8.140\t0\t0.5380\t5.9240\t94.10\t4.3996\t4\t307.0\t21.00\t394.33\t16.30\t15.60\n0.84054\t0.00\t8.140\t0\t0.5380\t5.5990\t85.70\t4.4546\t4\t307.0\t21.00\t303.42\t16.51\t13.90\n0.67191\t0.00\t8.140\t0\t0.5380\t5.8130\t90.30\t4.6820\t4\t307.0\t21.00\t376.88\t14.81\t16.60\n0.95577\t0.00\t8.140\t0\t0.5380\t6.0470\t88.80\t4.4534\t4\t307.0\t21.00\t306.38\t17.28\t14.80\n0.77299\t0.00\t8.140\t0\t0.5380\t6.4950\t94.40\t4.4547\t4\t307.0\t21.00\t387.94\t12.80\t18.40\n1.00245\t0.00\t8.140\t0\t0.5380\t6.6740\t87.30\t4.2390\t4\t307.0\t21.00\t380.23\t11.98\t21.00\n1.13081\t0.00\t8.140\t0\t0.5380\t5.7130\t94.10\t4.2330\t4\t307.0\t21.00\t360.17\t22.60\t12.70\n1.35472\t0.00\t8.140\t0\t0.5380\t6.0720\t100.00\t4.1750\t4\t307.0\t21.00\t376.73\t13.04\t14.50\n1.38799\t0.00\t8.140\t0\t0.5380\t5.9500\t82.00\t3.9900\t4\t307.0\t21.00\t232.60\t27.71\t13.20\n1.15172\t0.00\t8.140\t0\t0.5380\t5.7010\t95.00\t3.7872\t4\t307.0\t21.00\t358.77\t18.35\t13.10\n1.61282\t0.00\t8.140\t0\t0.5380\t6.0960\t96.90\t3.7598\t4\t307.0\t21.00\t248.31\t20.34\t13.50\n0.06417\t0.00\t5.960\t0\t0.4990\t5.9330\t68.20\t3.3603\t5\t279.0\t19.20\t396.90\t9.68\t18.90\n0.09744\t0.00\t5.960\t0\t0.4990\t5.8410\t61.40\t3.3779\t5\t279.0\t19.20\t377.56\t11.41\t20.00\n0.08014\t0.00\t5.960\t0\t0.4990\t5.8500\t41.50\t3.9342\t5\t279.0\t19.20\t396.90\t8.77\t21.00\n0.17505\t0.00\t5.960\t0\t0.4990\t5.9660\t30.20\t3.8473\t5\t279.0\t19.20\t393.43\t10.13\t24.70\n0.02763\t75.00\t2.950\t0\t0.4280\t6.5950\t21.80\t5.4011\t3\t252.0\t18.30\t395.63\t4.32\t30.80\n0.03359\t75.00\t2.950\t0\t0.4280\t7.0240\t15.80\t5.4011\t3\t252.0\t18.30\t395.62\t1.98\t34.90\n0.12744\t0.00\t6.910\t0\t0.4480\t6.7700\t2.90\t5.7209\t3\t233.0\t17.90\t385.41\t4.84\t26.60\n0.14150\t0.00\t6.910\t0\t0.4480\t6.1690\t6.60\t5.7209\t3\t233.0\t17.90\t383.37\t5.81\t25.30\n0.15936\t0.00\t6.910\t0\t0.4480\t6.2110\t6.50\t5.7209\t3\t233.0\t17.90\t394.46\t7.44\t24.70\n0.12269\t0.00\t6.910\t0\t0.4480\t6.0690\t40.00\t5.7209\t3\t233.0\t17.90\t389.39\t9.55\t21.20\n0.17142\t0.00\t6.910\t0\t0.4480\t5.6820\t33.80\t5.1004\t3\t233.0\t17.90\t396.90\t10.21\t19.30\n0.18836\t0.00\t6.910\t0\t0.4480\t5.7860\t33.30\t5.1004\t3\t233.0\t17.90\t396.90\t14.15\t20.00\n0.22927\t0.00\t6.910\t0\t0.4480\t6.0300\t85.50\t5.6894\t3\t233.0\t17.90\t392.74\t18.80\t16.60\n0.25387\t0.00\t6.910\t0\t0.4480\t5.3990\t95.30\t5.8700\t3\t233.0\t17.90\t396.90\t30.81\t14.40\n0.21977\t0.00\t6.910\t0\t0.4480\t5.6020\t62.00\t6.0877\t3\t233.0\t17.90\t396.90\t16.20\t19.40\n0.08873\t21.00\t5.640\t0\t0.4390\t5.9630\t45.70\t6.8147\t4\t243.0\t16.80\t395.56\t13.45\t19.70\n0.04337\t21.00\t5.640\t0\t0.4390\t6.1150\t63.00\t6.8147\t4\t243.0\t16.80\t393.97\t9.43\t20.50\n0.05360\t21.00\t5.640\t0\t0.4390\t6.5110\t21.10\t6.8147\t4\t243.0\t16.80\t396.90\t5.28\t25.00\n0.04981\t21.00\t5.640\t0\t0.4390\t5.9980\t21.40\t6.8147\t4\t243.0\t16.80\t396.90\t8.43\t23.40\n0.01360\t75.00\t4.000\t0\t0.4100\t5.8880\t47.60\t7.3197\t3\t469.0\t21.10\t396.90\t14.80\t18.90\n0.01311\t90.00\t1.220\t0\t0.4030\t7.2490\t21.90\t8.6966\t5\t226.0\t17.90\t395.93\t4.81\t35.40\n0.02055\t85.00\t0.740\t0\t0.4100\t6.3830\t35.70\t9.1876\t2\t313.0\t17.30\t396.90\t5.77\t24.70\n0.01432\t100.00\t1.320\t0\t0.4110\t6.8160\t40.50\t8.3248\t5\t256.0\t15.10\t392.90\t3.95\t31.60\n0.15445\t25.00\t5.130\t0\t0.4530\t6.1450\t29.20\t7.8148\t8\t284.0\t19.70\t390.68\t6.86\t23.30\n0.10328\t25.00\t5.130\t0\t0.4530\t5.9270\t47.20\t6.9320\t8\t284.0\t19.70\t396.90\t9.22\t19.60\n0.14932\t25.00\t5.130\t0\t0.4530\t5.7410\t66.20\t7.2254\t8\t284.0\t19.70\t395.11\t13.15\t18.70\n0.17171\t25.00\t5.130\t0\t0.4530\t5.9660\t93.40\t6.8185\t8\t284.0\t19.70\t378.08\t14.44\t16.00\n0.11027\t25.00\t5.130\t0\t0.4530\t6.4560\t67.80\t7.2255\t8\t284.0\t19.70\t396.90\t6.73\t22.20\n0.12650\t25.00\t5.130\t0\t0.4530\t6.7620\t43.40\t7.9809\t8\t284.0\t19.70\t395.58\t9.50\t25.00\n0.01951\t17.50\t1.380\t0\t0.4161\t7.1040\t59.50\t9.2229\t3\t216.0\t18.60\t393.24\t8.05\t33.00\n0.03584\t80.00\t3.370\t0\t0.3980\t6.2900\t17.80\t6.6115\t4\t337.0\t16.10\t396.90\t4.67\t23.50\n0.04379\t80.00\t3.370\t0\t0.3980\t5.7870\t31.10\t6.6115\t4\t337.0\t16.10\t396.90\t10.24\t19.40\n0.05789\t12.50\t6.070\t0\t0.4090\t5.8780\t21.40\t6.4980\t4\t345.0\t18.90\t396.21\t8.10\t22.00\n0.13554\t12.50\t6.070\t0\t0.4090\t5.5940\t36.80\t6.4980\t4\t345.0\t18.90\t396.90\t13.09\t17.40\n0.12816\t12.50\t6.070\t0\t0.4090\t5.8850\t33.00\t6.4980\t4\t345.0\t18.90\t396.90\t8.79\t20.90\n0.08826\t0.00\t10.810\t0\t0.4130\t6.4170\t6.60\t5.2873\t4\t305.0\t19.20\t383.73\t6.72\t24.20\n0.15876\t0.00\t10.810\t0\t0.4130\t5.9610\t17.50\t5.2873\t4\t305.0\t19.20\t376.94\t9.88\t21.70\n0.09164\t0.00\t10.810\t0\t0.4130\t6.0650\t7.80\t5.2873\t4\t305.0\t19.20\t390.91\t5.52\t22.80\n0.19539\t0.00\t10.810\t0\t0.4130\t6.2450\t6.20\t5.2873\t4\t305.0\t19.20\t377.17\t7.54\t23.40\n0.07896\t0.00\t12.830\t0\t0.4370\t6.2730\t6.00\t4.2515\t5\t398.0\t18.70\t394.92\t6.78\t24.10\n0.09512\t0.00\t12.830\t0\t0.4370\t6.2860\t45.00\t4.5026\t5\t398.0\t18.70\t383.23\t8.94\t21.40\n0.10153\t0.00\t12.830\t0\t0.4370\t6.2790\t74.50\t4.0522\t5\t398.0\t18.70\t373.66\t11.97\t20.00\n0.08707\t0.00\t12.830\t0\t0.4370\t6.1400\t45.80\t4.0905\t5\t398.0\t18.70\t386.96\t10.27\t20.80\n0.05646\t0.00\t12.830\t0\t0.4370\t6.2320\t53.70\t5.0141\t5\t398.0\t18.70\t386.40\t12.34\t21.20\n0.08387\t0.00\t12.830\t0\t0.4370\t5.8740\t36.60\t4.5026\t5\t398.0\t18.70\t396.06\t9.10\t20.30\n0.04113\t25.00\t4.860\t0\t0.4260\t6.7270\t33.50\t5.4007\t4\t281.0\t19.00\t396.90\t5.29\t28.00\n0.04462\t25.00\t4.860\t0\t0.4260\t6.6190\t70.40\t5.4007\t4\t281.0\t19.00\t395.63\t7.22\t23.90\n0.03659\t25.00\t4.860\t0\t0.4260\t6.3020\t32.20\t5.4007\t4\t281.0\t19.00\t396.90\t6.72\t24.80\n0.03551\t25.00\t4.860\t0\t0.4260\t6.1670\t46.70\t5.4007\t4\t281.0\t19.00\t390.64\t7.51\t22.90\n0.05059\t0.00\t4.490\t0\t0.4490\t6.3890\t48.00\t4.7794\t3\t247.0\t18.50\t396.90\t9.62\t23.90\n0.05735\t0.00\t4.490\t0\t0.4490\t6.6300\t56.10\t4.4377\t3\t247.0\t18.50\t392.30\t6.53\t26.60\n0.05188\t0.00\t4.490\t0\t0.4490\t6.0150\t45.10\t4.4272\t3\t247.0\t18.50\t395.99\t12.86\t22.50\n0.07151\t0.00\t4.490\t0\t0.4490\t6.1210\t56.80\t3.7476\t3\t247.0\t18.50\t395.15\t8.44\t22.20\n0.05660\t0.00\t3.410\t0\t0.4890\t7.0070\t86.30\t3.4217\t2\t270.0\t17.80\t396.90\t5.50\t23.60\n0.05302\t0.00\t3.410\t0\t0.4890\t7.0790\t63.10\t3.4145\t2\t270.0\t17.80\t396.06\t5.70\t28.70\n0.04684\t0.00\t3.410\t0\t0.4890\t6.4170\t66.10\t3.0923\t2\t270.0\t17.80\t392.18\t8.81\t22.60\n0.03932\t0.00\t3.410\t0\t0.4890\t6.4050\t73.90\t3.0921\t2\t270.0\t17.80\t393.55\t8.20\t22.00\n0.04203\t28.00\t15.040\t0\t0.4640\t6.4420\t53.60\t3.6659\t4\t270.0\t18.20\t395.01\t8.16\t22.90\n0.02875\t28.00\t15.040\t0\t0.4640\t6.2110\t28.90\t3.6659\t4\t270.0\t18.20\t396.33\t6.21\t25.00\n0.04294\t28.00\t15.040\t0\t0.4640\t6.2490\t77.30\t3.6150\t4\t270.0\t18.20\t396.90\t10.59\t20.60\n0.12204\t0.00\t2.890\t0\t0.4450\t6.6250\t57.80\t3.4952\t2\t276.0\t18.00\t357.98\t6.65\t28.40\n0.11504\t0.00\t2.890\t0\t0.4450\t6.1630\t69.60\t3.4952\t2\t276.0\t18.00\t391.83\t11.34\t21.40\n0.12083\t0.00\t2.890\t0\t0.4450\t8.0690\t76.00\t3.4952\t2\t276.0\t18.00\t396.90\t4.21\t38.70\n0.08187\t0.00\t2.890\t0\t0.4450\t7.8200\t36.90\t3.4952\t2\t276.0\t18.00\t393.53\t3.57\t43.80\n0.06860\t0.00\t2.890\t0\t0.4450\t7.4160\t62.50\t3.4952\t2\t276.0\t18.00\t396.90\t6.19\t33.20\n0.14866\t0.00\t8.560\t0\t0.5200\t6.7270\t79.90\t2.7778\t5\t384.0\t20.90\t394.76\t9.42\t27.50\n0.11432\t0.00\t8.560\t0\t0.5200\t6.7810\t71.30\t2.8561\t5\t384.0\t20.90\t395.58\t7.67\t26.50\n0.22876\t0.00\t8.560\t0\t0.5200\t6.4050\t85.40\t2.7147\t5\t384.0\t20.90\t70.80\t10.63\t18.60\n0.21161\t0.00\t8.560\t0\t0.5200\t6.1370\t87.40\t2.7147\t5\t384.0\t20.90\t394.47\t13.44\t19.30\n0.13960\t0.00\t8.560\t0\t0.5200\t6.1670\t90.00\t2.4210\t5\t384.0\t20.90\t392.69\t12.33\t20.10\n0.13262\t0.00\t8.560\t0\t0.5200\t5.8510\t96.70\t2.1069\t5\t384.0\t20.90\t394.05\t16.47\t19.50\n0.17120\t0.00\t8.560\t0\t0.5200\t5.8360\t91.90\t2.2110\t5\t384.0\t20.90\t395.67\t18.66\t19.50\n0.13117\t0.00\t8.560\t0\t0.5200\t6.1270\t85.20\t2.1224\t5\t384.0\t20.90\t387.69\t14.09\t20.40\n0.12802\t0.00\t8.560\t0\t0.5200\t6.4740\t97.10\t2.4329\t5\t384.0\t20.90\t395.24\t12.27\t19.80\n0.26363\t0.00\t8.560\t0\t0.5200\t6.2290\t91.20\t2.5451\t5\t384.0\t20.90\t391.23\t15.55\t19.40\n0.10793\t0.00\t8.560\t0\t0.5200\t6.1950\t54.40\t2.7778\t5\t384.0\t20.90\t393.49\t13.00\t21.70\n0.10084\t0.00\t10.010\t0\t0.5470\t6.7150\t81.60\t2.6775\t6\t432.0\t17.80\t395.59\t10.16\t22.80\n0.12329\t0.00\t10.010\t0\t0.5470\t5.9130\t92.90\t2.3534\t6\t432.0\t17.80\t394.95\t16.21\t18.80\n0.22212\t0.00\t10.010\t0\t0.5470\t6.0920\t95.40\t2.5480\t6\t432.0\t17.80\t396.90\t17.09\t18.70\n0.14231\t0.00\t10.010\t0\t0.5470\t6.2540\t84.20\t2.2565\t6\t432.0\t17.80\t388.74\t10.45\t18.50\n0.17134\t0.00\t10.010\t0\t0.5470\t5.9280\t88.20\t2.4631\t6\t432.0\t17.80\t344.91\t15.76\t18.30\n0.13158\t0.00\t10.010\t0\t0.5470\t6.1760\t72.50\t2.7301\t6\t432.0\t17.80\t393.30\t12.04\t21.20\n0.15098\t0.00\t10.010\t0\t0.5470\t6.0210\t82.60\t2.7474\t6\t432.0\t17.80\t394.51\t10.30\t19.20\n0.13058\t0.00\t10.010\t0\t0.5470\t5.8720\t73.10\t2.4775\t6\t432.0\t17.80\t338.63\t15.37\t20.40\n0.14476\t0.00\t10.010\t0\t0.5470\t5.7310\t65.20\t2.7592\t6\t432.0\t17.80\t391.50\t13.61\t19.30\n0.06899\t0.00\t25.650\t0\t0.5810\t5.8700\t69.70\t2.2577\t2\t188.0\t19.10\t389.15\t14.37\t22.00\n0.07165\t0.00\t25.650\t0\t0.5810\t6.0040\t84.10\t2.1974\t2\t188.0\t19.10\t377.67\t14.27\t20.30\n0.09299\t0.00\t25.650\t0\t0.5810\t5.9610\t92.90\t2.0869\t2\t188.0\t19.10\t378.09\t17.93\t20.50\n0.15038\t0.00\t25.650\t0\t0.5810\t5.8560\t97.00\t1.9444\t2\t188.0\t19.10\t370.31\t25.41\t17.30\n0.09849\t0.00\t25.650\t0\t0.5810\t5.8790\t95.80\t2.0063\t2\t188.0\t19.10\t379.38\t17.58\t18.80\n0.16902\t0.00\t25.650\t0\t0.5810\t5.9860\t88.40\t1.9929\t2\t188.0\t19.10\t385.02\t14.81\t21.40\n0.38735\t0.00\t25.650\t0\t0.5810\t5.6130\t95.60\t1.7572\t2\t188.0\t19.10\t359.29\t27.26\t15.70\n0.25915\t0.00\t21.890\t0\t0.6240\t5.6930\t96.00\t1.7883\t4\t437.0\t21.20\t392.11\t17.19\t16.20\n0.32543\t0.00\t21.890\t0\t0.6240\t6.4310\t98.80\t1.8125\t4\t437.0\t21.20\t396.90\t15.39\t18.00\n0.88125\t0.00\t21.890\t0\t0.6240\t5.6370\t94.70\t1.9799\t4\t437.0\t21.20\t396.90\t18.34\t14.30\n0.34006\t0.00\t21.890\t0\t0.6240\t6.4580\t98.90\t2.1185\t4\t437.0\t21.20\t395.04\t12.60\t19.20\n1.19294\t0.00\t21.890\t0\t0.6240\t6.3260\t97.70\t2.2710\t4\t437.0\t21.20\t396.90\t12.26\t19.60\n0.59005\t0.00\t21.890\t0\t0.6240\t6.3720\t97.90\t2.3274\t4\t437.0\t21.20\t385.76\t11.12\t23.00\n0.32982\t0.00\t21.890\t0\t0.6240\t5.8220\t95.40\t2.4699\t4\t437.0\t21.20\t388.69\t15.03\t18.40\n0.97617\t0.00\t21.890\t0\t0.6240\t5.7570\t98.40\t2.3460\t4\t437.0\t21.20\t262.76\t17.31\t15.60\n0.55778\t0.00\t21.890\t0\t0.6240\t6.3350\t98.20\t2.1107\t4\t437.0\t21.20\t394.67\t16.96\t18.10\n0.32264\t0.00\t21.890\t0\t0.6240\t5.9420\t93.50\t1.9669\t4\t437.0\t21.20\t378.25\t16.90\t17.40\n0.35233\t0.00\t21.890\t0\t0.6240\t6.4540\t98.40\t1.8498\t4\t437.0\t21.20\t394.08\t14.59\t17.10\n0.24980\t0.00\t21.890\t0\t0.6240\t5.8570\t98.20\t1.6686\t4\t437.0\t21.20\t392.04\t21.32\t13.30\n0.54452\t0.00\t21.890\t0\t0.6240\t6.1510\t97.90\t1.6687\t4\t437.0\t21.20\t396.90\t18.46\t17.80\n0.29090\t0.00\t21.890\t0\t0.6240\t6.1740\t93.60\t1.6119\t4\t437.0\t21.20\t388.08\t24.16\t14.00\n1.62864\t0.00\t21.890\t0\t0.6240\t5.0190\t100.00\t1.4394\t4\t437.0\t21.20\t396.90\t34.41\t14.40\n3.32105\t0.00\t19.580\t1\t0.8710\t5.4030\t100.00\t1.3216\t5\t403.0\t14.70\t396.90\t26.82\t13.40\n4.09740\t0.00\t19.580\t0\t0.8710\t5.4680\t100.00\t1.4118\t5\t403.0\t14.70\t396.90\t26.42\t15.60\n2.77974\t0.00\t19.580\t0\t0.8710\t4.9030\t97.80\t1.3459\t5\t403.0\t14.70\t396.90\t29.29\t11.80\n2.37934\t0.00\t19.580\t0\t0.8710\t6.1300\t100.00\t1.4191\t5\t403.0\t14.70\t172.91\t27.80\t13.80\n2.15505\t0.00\t19.580\t0\t0.8710\t5.6280\t100.00\t1.5166\t5\t403.0\t14.70\t169.27\t16.65\t15.60\n2.36862\t0.00\t19.580\t0\t0.8710\t4.9260\t95.70\t1.4608\t5\t403.0\t14.70\t391.71\t29.53\t14.60\n2.33099\t0.00\t19.580\t0\t0.8710\t5.1860\t93.80\t1.5296\t5\t403.0\t14.70\t356.99\t28.32\t17.80\n2.73397\t0.00\t19.580\t0\t0.8710\t5.5970\t94.90\t1.5257\t5\t403.0\t14.70\t351.85\t21.45\t15.40\n1.65660\t0.00\t19.580\t0\t0.8710\t6.1220\t97.30\t1.6180\t5\t403.0\t14.70\t372.80\t14.10\t21.50\n1.49632\t0.00\t19.580\t0\t0.8710\t5.4040\t100.00\t1.5916\t5\t403.0\t14.70\t341.60\t13.28\t19.60\n1.12658\t0.00\t19.580\t1\t0.8710\t5.0120\t88.00\t1.6102\t5\t403.0\t14.70\t343.28\t12.12\t15.30\n2.14918\t0.00\t19.580\t0\t0.8710\t5.7090\t98.50\t1.6232\t5\t403.0\t14.70\t261.95\t15.79\t19.40\n1.41385\t0.00\t19.580\t1\t0.8710\t6.1290\t96.00\t1.7494\t5\t403.0\t14.70\t321.02\t15.12\t17.00\n3.53501\t0.00\t19.580\t1\t0.8710\t6.1520\t82.60\t1.7455\t5\t403.0\t14.70\t88.01\t15.02\t15.60\n2.44668\t0.00\t19.580\t0\t0.8710\t5.2720\t94.00\t1.7364\t5\t403.0\t14.70\t88.63\t16.14\t13.10\n1.22358\t0.00\t19.580\t0\t0.6050\t6.9430\t97.40\t1.8773\t5\t403.0\t14.70\t363.43\t4.59\t41.30\n1.34284\t0.00\t19.580\t0\t0.6050\t6.0660\t100.00\t1.7573\t5\t403.0\t14.70\t353.89\t6.43\t24.30\n1.42502\t0.00\t19.580\t0\t0.8710\t6.5100\t100.00\t1.7659\t5\t403.0\t14.70\t364.31\t7.39\t23.30\n1.27346\t0.00\t19.580\t1\t0.6050\t6.2500\t92.60\t1.7984\t5\t403.0\t14.70\t338.92\t5.50\t27.00\n1.46336\t0.00\t19.580\t0\t0.6050\t7.4890\t90.80\t1.9709\t5\t403.0\t14.70\t374.43\t1.73\t50.00\n1.83377\t0.00\t19.580\t1\t0.6050\t7.8020\t98.20\t2.0407\t5\t403.0\t14.70\t389.61\t1.92\t50.00\n1.51902\t0.00\t19.580\t1\t0.6050\t8.3750\t93.90\t2.1620\t5\t403.0\t14.70\t388.45\t3.32\t50.00\n2.24236\t0.00\t19.580\t0\t0.6050\t5.8540\t91.80\t2.4220\t5\t403.0\t14.70\t395.11\t11.64\t22.70\n2.92400\t0.00\t19.580\t0\t0.6050\t6.1010\t93.00\t2.2834\t5\t403.0\t14.70\t240.16\t9.81\t25.00\n2.01019\t0.00\t19.580\t0\t0.6050\t7.9290\t96.20\t2.0459\t5\t403.0\t14.70\t369.30\t3.70\t50.00\n1.80028\t0.00\t19.580\t0\t0.6050\t5.8770\t79.20\t2.4259\t5\t403.0\t14.70\t227.61\t12.14\t23.80\n2.30040\t0.00\t19.580\t0\t0.6050\t6.3190\t96.10\t2.1000\t5\t403.0\t14.70\t297.09\t11.10\t23.80\n2.44953\t0.00\t19.580\t0\t0.6050\t6.4020\t95.20\t2.2625\t5\t403.0\t14.70\t330.04\t11.32\t22.30\n1.20742\t0.00\t19.580\t0\t0.6050\t5.8750\t94.60\t2.4259\t5\t403.0\t14.70\t292.29\t14.43\t17.40\n2.31390\t0.00\t19.580\t0\t0.6050\t5.8800\t97.30\t2.3887\t5\t403.0\t14.70\t348.13\t12.03\t19.10\n0.13914\t0.00\t4.050\t0\t0.5100\t5.5720\t88.50\t2.5961\t5\t296.0\t16.60\t396.90\t14.69\t23.10\n0.09178\t0.00\t4.050\t0\t0.5100\t6.4160\t84.10\t2.6463\t5\t296.0\t16.60\t395.50\t9.04\t23.60\n0.08447\t0.00\t4.050\t0\t0.5100\t5.8590\t68.70\t2.7019\t5\t296.0\t16.60\t393.23\t9.64\t22.60\n0.06664\t0.00\t4.050\t0\t0.5100\t6.5460\t33.10\t3.1323\t5\t296.0\t16.60\t390.96\t5.33\t29.40\n0.07022\t0.00\t4.050\t0\t0.5100\t6.0200\t47.20\t3.5549\t5\t296.0\t16.60\t393.23\t10.11\t23.20\n0.05425\t0.00\t4.050\t0\t0.5100\t6.3150\t73.40\t3.3175\t5\t296.0\t16.60\t395.60\t6.29\t24.60\n0.06642\t0.00\t4.050\t0\t0.5100\t6.8600\t74.40\t2.9153\t5\t296.0\t16.60\t391.27\t6.92\t29.90\n0.05780\t0.00\t2.460\t0\t0.4880\t6.9800\t58.40\t2.8290\t3\t193.0\t17.80\t396.90\t5.04\t37.20\n0.06588\t0.00\t2.460\t0\t0.4880\t7.7650\t83.30\t2.7410\t3\t193.0\t17.80\t395.56\t7.56\t39.80\n0.06888\t0.00\t2.460\t0\t0.4880\t6.1440\t62.20\t2.5979\t3\t193.0\t17.80\t396.90\t9.45\t36.20\n0.09103\t0.00\t2.460\t0\t0.4880\t7.1550\t92.20\t2.7006\t3\t193.0\t17.80\t394.12\t4.82\t37.90\n0.10008\t0.00\t2.460\t0\t0.4880\t6.5630\t95.60\t2.8470\t3\t193.0\t17.80\t396.90\t5.68\t32.50\n0.08308\t0.00\t2.460\t0\t0.4880\t5.6040\t89.80\t2.9879\t3\t193.0\t17.80\t391.00\t13.98\t26.40\n0.06047\t0.00\t2.460\t0\t0.4880\t6.1530\t68.80\t3.2797\t3\t193.0\t17.80\t387.11\t13.15\t29.60\n0.05602\t0.00\t2.460\t0\t0.4880\t7.8310\t53.60\t3.1992\t3\t193.0\t17.80\t392.63\t4.45\t50.00\n0.07875\t45.00\t3.440\t0\t0.4370\t6.7820\t41.10\t3.7886\t5\t398.0\t15.20\t393.87\t6.68\t32.00\n0.12579\t45.00\t3.440\t0\t0.4370\t6.5560\t29.10\t4.5667\t5\t398.0\t15.20\t382.84\t4.56\t29.80\n0.08370\t45.00\t3.440\t0\t0.4370\t7.1850\t38.90\t4.5667\t5\t398.0\t15.20\t396.90\t5.39\t34.90\n0.09068\t45.00\t3.440\t0\t0.4370\t6.9510\t21.50\t6.4798\t5\t398.0\t15.20\t377.68\t5.10\t37.00\n0.06911\t45.00\t3.440\t0\t0.4370\t6.7390\t30.80\t6.4798\t5\t398.0\t15.20\t389.71\t4.69\t30.50\n0.08664\t45.00\t3.440\t0\t0.4370\t7.1780\t26.30\t6.4798\t5\t398.0\t15.20\t390.49\t2.87\t36.40\n0.02187\t60.00\t2.930\t0\t0.4010\t6.8000\t9.90\t6.2196\t1\t265.0\t15.60\t393.37\t5.03\t31.10\n0.01439\t60.00\t2.930\t0\t0.4010\t6.6040\t18.80\t6.2196\t1\t265.0\t15.60\t376.70\t4.38\t29.10\n0.01381\t80.00\t0.460\t0\t0.4220\t7.8750\t32.00\t5.6484\t4\t255.0\t14.40\t394.23\t2.97\t50.00\n0.04011\t80.00\t1.520\t0\t0.4040\t7.2870\t34.10\t7.3090\t2\t329.0\t12.60\t396.90\t4.08\t33.30\n0.04666\t80.00\t1.520\t0\t0.4040\t7.1070\t36.60\t7.3090\t2\t329.0\t12.60\t354.31\t8.61\t30.30\n0.03768\t80.00\t1.520\t0\t0.4040\t7.2740\t38.30\t7.3090\t2\t329.0\t12.60\t392.20\t6.62\t34.60\n0.03150\t95.00\t1.470\t0\t0.4030\t6.9750\t15.30\t7.6534\t3\t402.0\t17.00\t396.90\t4.56\t34.90\n0.01778\t95.00\t1.470\t0\t0.4030\t7.1350\t13.90\t7.6534\t3\t402.0\t17.00\t384.30\t4.45\t32.90\n0.03445\t82.50\t2.030\t0\t0.4150\t6.1620\t38.40\t6.2700\t2\t348.0\t14.70\t393.77\t7.43\t24.10\n0.02177\t82.50\t2.030\t0\t0.4150\t7.6100\t15.70\t6.2700\t2\t348.0\t14.70\t395.38\t3.11\t42.30\n0.03510\t95.00\t2.680\t0\t0.4161\t7.8530\t33.20\t5.1180\t4\t224.0\t14.70\t392.78\t3.81\t48.50\n0.02009\t95.00\t2.680\t0\t0.4161\t8.0340\t31.90\t5.1180\t4\t224.0\t14.70\t390.55\t2.88\t50.00\n0.13642\t0.00\t10.590\t0\t0.4890\t5.8910\t22.30\t3.9454\t4\t277.0\t18.60\t396.90\t10.87\t22.60\n0.22969\t0.00\t10.590\t0\t0.4890\t6.3260\t52.50\t4.3549\t4\t277.0\t18.60\t394.87\t10.97\t24.40\n0.25199\t0.00\t10.590\t0\t0.4890\t5.7830\t72.70\t4.3549\t4\t277.0\t18.60\t389.43\t18.06\t22.50\n0.13587\t0.00\t10.590\t1\t0.4890\t6.0640\t59.10\t4.2392\t4\t277.0\t18.60\t381.32\t14.66\t24.40\n0.43571\t0.00\t10.590\t1\t0.4890\t5.3440\t100.00\t3.8750\t4\t277.0\t18.60\t396.90\t23.09\t20.00\n0.17446\t0.00\t10.590\t1\t0.4890\t5.9600\t92.10\t3.8771\t4\t277.0\t18.60\t393.25\t17.27\t21.70\n0.37578\t0.00\t10.590\t1\t0.4890\t5.4040\t88.60\t3.6650\t4\t277.0\t18.60\t395.24\t23.98\t19.30\n0.21719\t0.00\t10.590\t1\t0.4890\t5.8070\t53.80\t3.6526\t4\t277.0\t18.60\t390.94\t16.03\t22.40\n0.14052\t0.00\t10.590\t0\t0.4890\t6.3750\t32.30\t3.9454\t4\t277.0\t18.60\t385.81\t9.38\t28.10\n0.28955\t0.00\t10.590\t0\t0.4890\t5.4120\t9.80\t3.5875\t4\t277.0\t18.60\t348.93\t29.55\t23.70\n0.19802\t0.00\t10.590\t0\t0.4890\t6.1820\t42.40\t3.9454\t4\t277.0\t18.60\t393.63\t9.47\t25.00\n0.04560\t0.00\t13.890\t1\t0.5500\t5.8880\t56.00\t3.1121\t5\t276.0\t16.40\t392.80\t13.51\t23.30\n0.07013\t0.00\t13.890\t0\t0.5500\t6.6420\t85.10\t3.4211\t5\t276.0\t16.40\t392.78\t9.69\t28.70\n0.11069\t0.00\t13.890\t1\t0.5500\t5.9510\t93.80\t2.8893\t5\t276.0\t16.40\t396.90\t17.92\t21.50\n0.11425\t0.00\t13.890\t1\t0.5500\t6.3730\t92.40\t3.3633\t5\t276.0\t16.40\t393.74\t10.50\t23.00\n0.35809\t0.00\t6.200\t1\t0.5070\t6.9510\t88.50\t2.8617\t8\t307.0\t17.40\t391.70\t9.71\t26.70\n0.40771\t0.00\t6.200\t1\t0.5070\t6.1640\t91.30\t3.0480\t8\t307.0\t17.40\t395.24\t21.46\t21.70\n0.62356\t0.00\t6.200\t1\t0.5070\t6.8790\t77.70\t3.2721\t8\t307.0\t17.40\t390.39\t9.93\t27.50\n0.61470\t0.00\t6.200\t0\t0.5070\t6.6180\t80.80\t3.2721\t8\t307.0\t17.40\t396.90\t7.60\t30.10\n0.31533\t0.00\t6.200\t0\t0.5040\t8.2660\t78.30\t2.8944\t8\t307.0\t17.40\t385.05\t4.14\t44.80\n0.52693\t0.00\t6.200\t0\t0.5040\t8.7250\t83.00\t2.8944\t8\t307.0\t17.40\t382.00\t4.63\t50.00\n0.38214\t0.00\t6.200\t0\t0.5040\t8.0400\t86.50\t3.2157\t8\t307.0\t17.40\t387.38\t3.13\t37.60\n0.41238\t0.00\t6.200\t0\t0.5040\t7.1630\t79.90\t3.2157\t8\t307.0\t17.40\t372.08\t6.36\t31.60\n0.29819\t0.00\t6.200\t0\t0.5040\t7.6860\t17.00\t3.3751\t8\t307.0\t17.40\t377.51\t3.92\t46.70\n0.44178\t0.00\t6.200\t0\t0.5040\t6.5520\t21.40\t3.3751\t8\t307.0\t17.40\t380.34\t3.76\t31.50\n0.53700\t0.00\t6.200\t0\t0.5040\t5.9810\t68.10\t3.6715\t8\t307.0\t17.40\t378.35\t11.65\t24.30\n0.46296\t0.00\t6.200\t0\t0.5040\t7.4120\t76.90\t3.6715\t8\t307.0\t17.40\t376.14\t5.25\t31.70\n0.57529\t0.00\t6.200\t0\t0.5070\t8.3370\t73.30\t3.8384\t8\t307.0\t17.40\t385.91\t2.47\t41.70\n0.33147\t0.00\t6.200\t0\t0.5070\t8.2470\t70.40\t3.6519\t8\t307.0\t17.40\t378.95\t3.95\t48.30\n0.44791\t0.00\t6.200\t1\t0.5070\t6.7260\t66.50\t3.6519\t8\t307.0\t17.40\t360.20\t8.05\t29.00\n0.33045\t0.00\t6.200\t0\t0.5070\t6.0860\t61.50\t3.6519\t8\t307.0\t17.40\t376.75\t10.88\t24.00\n0.52058\t0.00\t6.200\t1\t0.5070\t6.6310\t76.50\t4.1480\t8\t307.0\t17.40\t388.45\t9.54\t25.10\n0.51183\t0.00\t6.200\t0\t0.5070\t7.3580\t71.60\t4.1480\t8\t307.0\t17.40\t390.07\t4.73\t31.50\n0.08244\t30.00\t4.930\t0\t0.4280\t6.4810\t18.50\t6.1899\t6\t300.0\t16.60\t379.41\t6.36\t23.70\n0.09252\t30.00\t4.930\t0\t0.4280\t6.6060\t42.20\t6.1899\t6\t300.0\t16.60\t383.78\t7.37\t23.30\n0.11329\t30.00\t4.930\t0\t0.4280\t6.8970\t54.30\t6.3361\t6\t300.0\t16.60\t391.25\t11.38\t22.00\n0.10612\t30.00\t4.930\t0\t0.4280\t6.0950\t65.10\t6.3361\t6\t300.0\t16.60\t394.62\t12.40\t20.10\n0.10290\t30.00\t4.930\t0\t0.4280\t6.3580\t52.90\t7.0355\t6\t300.0\t16.60\t372.75\t11.22\t22.20\n0.12757\t30.00\t4.930\t0\t0.4280\t6.3930\t7.80\t7.0355\t6\t300.0\t16.60\t374.71\t5.19\t23.70\n0.20608\t22.00\t5.860\t0\t0.4310\t5.5930\t76.50\t7.9549\t7\t330.0\t19.10\t372.49\t12.50\t17.60\n0.19133\t22.00\t5.860\t0\t0.4310\t5.6050\t70.20\t7.9549\t7\t330.0\t19.10\t389.13\t18.46\t18.50\n0.33983\t22.00\t5.860\t0\t0.4310\t6.1080\t34.90\t8.0555\t7\t330.0\t19.10\t390.18\t9.16\t24.30\n0.19657\t22.00\t5.860\t0\t0.4310\t6.2260\t79.20\t8.0555\t7\t330.0\t19.10\t376.14\t10.15\t20.50\n0.16439\t22.00\t5.860\t0\t0.4310\t6.4330\t49.10\t7.8265\t7\t330.0\t19.10\t374.71\t9.52\t24.50\n0.19073\t22.00\t5.860\t0\t0.4310\t6.7180\t17.50\t7.8265\t7\t330.0\t19.10\t393.74\t6.56\t26.20\n0.14030\t22.00\t5.860\t0\t0.4310\t6.4870\t13.00\t7.3967\t7\t330.0\t19.10\t396.28\t5.90\t24.40\n0.21409\t22.00\t5.860\t0\t0.4310\t6.4380\t8.90\t7.3967\t7\t330.0\t19.10\t377.07\t3.59\t24.80\n0.08221\t22.00\t5.860\t0\t0.4310\t6.9570\t6.80\t8.9067\t7\t330.0\t19.10\t386.09\t3.53\t29.60\n0.36894\t22.00\t5.860\t0\t0.4310\t8.2590\t8.40\t8.9067\t7\t330.0\t19.10\t396.90\t3.54\t42.80\n0.04819\t80.00\t3.640\t0\t0.3920\t6.1080\t32.00\t9.2203\t1\t315.0\t16.40\t392.89\t6.57\t21.90\n0.03548\t80.00\t3.640\t0\t0.3920\t5.8760\t19.10\t9.2203\t1\t315.0\t16.40\t395.18\t9.25\t20.90\n0.01538\t90.00\t3.750\t0\t0.3940\t7.4540\t34.20\t6.3361\t3\t244.0\t15.90\t386.34\t3.11\t44.00\n0.61154\t20.00\t3.970\t0\t0.6470\t8.7040\t86.90\t1.8010\t5\t264.0\t13.00\t389.70\t5.12\t50.00\n0.66351\t20.00\t3.970\t0\t0.6470\t7.3330\t100.00\t1.8946\t5\t264.0\t13.00\t383.29\t7.79\t36.00\n0.65665\t20.00\t3.970\t0\t0.6470\t6.8420\t100.00\t2.0107\t5\t264.0\t13.00\t391.93\t6.90\t30.10\n0.54011\t20.00\t3.970\t0\t0.6470\t7.2030\t81.80\t2.1121\t5\t264.0\t13.00\t392.80\t9.59\t33.80\n0.53412\t20.00\t3.970\t0\t0.6470\t7.5200\t89.40\t2.1398\t5\t264.0\t13.00\t388.37\t7.26\t43.10\n0.52014\t20.00\t3.970\t0\t0.6470\t8.3980\t91.50\t2.2885\t5\t264.0\t13.00\t386.86\t5.91\t48.80\n0.82526\t20.00\t3.970\t0\t0.6470\t7.3270\t94.50\t2.0788\t5\t264.0\t13.00\t393.42\t11.25\t31.00\n0.55007\t20.00\t3.970\t0\t0.6470\t7.2060\t91.60\t1.9301\t5\t264.0\t13.00\t387.89\t8.10\t36.50\n0.76162\t20.00\t3.970\t0\t0.6470\t5.5600\t62.80\t1.9865\t5\t264.0\t13.00\t392.40\t10.45\t22.80\n0.78570\t20.00\t3.970\t0\t0.6470\t7.0140\t84.60\t2.1329\t5\t264.0\t13.00\t384.07\t14.79\t30.70\n0.57834\t20.00\t3.970\t0\t0.5750\t8.2970\t67.00\t2.4216\t5\t264.0\t13.00\t384.54\t7.44\t50.00\n0.54050\t20.00\t3.970\t0\t0.5750\t7.4700\t52.60\t2.8720\t5\t264.0\t13.00\t390.30\t3.16\t43.50\n0.09065\t20.00\t6.960\t1\t0.4640\t5.9200\t61.50\t3.9175\t3\t223.0\t18.60\t391.34\t13.65\t20.70\n0.29916\t20.00\t6.960\t0\t0.4640\t5.8560\t42.10\t4.4290\t3\t223.0\t18.60\t388.65\t13.00\t21.10\n0.16211\t20.00\t6.960\t0\t0.4640\t6.2400\t16.30\t4.4290\t3\t223.0\t18.60\t396.90\t6.59\t25.20\n0.11460\t20.00\t6.960\t0\t0.4640\t6.5380\t58.70\t3.9175\t3\t223.0\t18.60\t394.96\t7.73\t24.40\n0.22188\t20.00\t6.960\t1\t0.4640\t7.6910\t51.80\t4.3665\t3\t223.0\t18.60\t390.77\t6.58\t35.20\n0.05644\t40.00\t6.410\t1\t0.4470\t6.7580\t32.90\t4.0776\t4\t254.0\t17.60\t396.90\t3.53\t32.40\n0.09604\t40.00\t6.410\t0\t0.4470\t6.8540\t42.80\t4.2673\t4\t254.0\t17.60\t396.90\t2.98\t32.00\n0.10469\t40.00\t6.410\t1\t0.4470\t7.2670\t49.00\t4.7872\t4\t254.0\t17.60\t389.25\t6.05\t33.20\n0.06127\t40.00\t6.410\t1\t0.4470\t6.8260\t27.60\t4.8628\t4\t254.0\t17.60\t393.45\t4.16\t33.10\n0.07978\t40.00\t6.410\t0\t0.4470\t6.4820\t32.10\t4.1403\t4\t254.0\t17.60\t396.90\t7.19\t29.10\n0.21038\t20.00\t3.330\t0\t0.4429\t6.8120\t32.20\t4.1007\t5\t216.0\t14.90\t396.90\t4.85\t35.10\n0.03578\t20.00\t3.330\t0\t0.4429\t7.8200\t64.50\t4.6947\t5\t216.0\t14.90\t387.31\t3.76\t45.40\n0.03705\t20.00\t3.330\t0\t0.4429\t6.9680\t37.20\t5.2447\t5\t216.0\t14.90\t392.23\t4.59\t35.40\n0.06129\t20.00\t3.330\t1\t0.4429\t7.6450\t49.70\t5.2119\t5\t216.0\t14.90\t377.07\t3.01\t46.00\n0.01501\t90.00\t1.210\t1\t0.4010\t7.9230\t24.80\t5.8850\t1\t198.0\t13.60\t395.52\t3.16\t50.00\n0.00906\t90.00\t2.970\t0\t0.4000\t7.0880\t20.80\t7.3073\t1\t285.0\t15.30\t394.72\t7.85\t32.20\n0.01096\t55.00\t2.250\t0\t0.3890\t6.4530\t31.90\t7.3073\t1\t300.0\t15.30\t394.72\t8.23\t22.00\n0.01965\t80.00\t1.760\t0\t0.3850\t6.2300\t31.50\t9.0892\t1\t241.0\t18.20\t341.60\t12.93\t20.10\n0.03871\t52.50\t5.320\t0\t0.4050\t6.2090\t31.30\t7.3172\t6\t293.0\t16.60\t396.90\t7.14\t23.20\n0.04590\t52.50\t5.320\t0\t0.4050\t6.3150\t45.60\t7.3172\t6\t293.0\t16.60\t396.90\t7.60\t22.30\n0.04297\t52.50\t5.320\t0\t0.4050\t6.5650\t22.90\t7.3172\t6\t293.0\t16.60\t371.72\t9.51\t24.80\n0.03502\t80.00\t4.950\t0\t0.4110\t6.8610\t27.90\t5.1167\t4\t245.0\t19.20\t396.90\t3.33\t28.50\n0.07886\t80.00\t4.950\t0\t0.4110\t7.1480\t27.70\t5.1167\t4\t245.0\t19.20\t396.90\t3.56\t37.30\n0.03615\t80.00\t4.950\t0\t0.4110\t6.6300\t23.40\t5.1167\t4\t245.0\t19.20\t396.90\t4.70\t27.90\n0.08265\t0.00\t13.920\t0\t0.4370\t6.1270\t18.40\t5.5027\t4\t289.0\t16.00\t396.90\t8.58\t23.90\n0.08199\t0.00\t13.920\t0\t0.4370\t6.0090\t42.30\t5.5027\t4\t289.0\t16.00\t396.90\t10.40\t21.70\n0.12932\t0.00\t13.920\t0\t0.4370\t6.6780\t31.10\t5.9604\t4\t289.0\t16.00\t396.90\t6.27\t28.60\n0.05372\t0.00\t13.920\t0\t0.4370\t6.5490\t51.00\t5.9604\t4\t289.0\t16.00\t392.85\t7.39\t27.10\n0.14103\t0.00\t13.920\t0\t0.4370\t5.7900\t58.00\t6.3200\t4\t289.0\t16.00\t396.90\t15.84\t20.30\n0.06466\t70.00\t2.240\t0\t0.4000\t6.3450\t20.10\t7.8278\t5\t358.0\t14.80\t368.24\t4.97\t22.50\n0.05561\t70.00\t2.240\t0\t0.4000\t7.0410\t10.00\t7.8278\t5\t358.0\t14.80\t371.58\t4.74\t29.00\n0.04417\t70.00\t2.240\t0\t0.4000\t6.8710\t47.40\t7.8278\t5\t358.0\t14.80\t390.86\t6.07\t24.80\n0.03537\t34.00\t6.090\t0\t0.4330\t6.5900\t40.40\t5.4917\t7\t329.0\t16.10\t395.75\t9.50\t22.00\n0.09266\t34.00\t6.090\t0\t0.4330\t6.4950\t18.40\t5.4917\t7\t329.0\t16.10\t383.61\t8.67\t26.40\n0.10000\t34.00\t6.090\t0\t0.4330\t6.9820\t17.70\t5.4917\t7\t329.0\t16.10\t390.43\t4.86\t33.10\n0.05515\t33.00\t2.180\t0\t0.4720\t7.2360\t41.10\t4.0220\t7\t222.0\t18.40\t393.68\t6.93\t36.10\n0.05479\t33.00\t2.180\t0\t0.4720\t6.6160\t58.10\t3.3700\t7\t222.0\t18.40\t393.36\t8.93\t28.40\n0.07503\t33.00\t2.180\t0\t0.4720\t7.4200\t71.90\t3.0992\t7\t222.0\t18.40\t396.90\t6.47\t33.40\n0.04932\t33.00\t2.180\t0\t0.4720\t6.8490\t70.30\t3.1827\t7\t222.0\t18.40\t396.90\t7.53\t28.20\n0.49298\t0.00\t9.900\t0\t0.5440\t6.6350\t82.50\t3.3175\t4\t304.0\t18.40\t396.90\t4.54\t22.80\n0.34940\t0.00\t9.900\t0\t0.5440\t5.9720\t76.70\t3.1025\t4\t304.0\t18.40\t396.24\t9.97\t20.30\n2.63548\t0.00\t9.900\t0\t0.5440\t4.9730\t37.80\t2.5194\t4\t304.0\t18.40\t350.45\t12.64\t16.10\n0.79041\t0.00\t9.900\t0\t0.5440\t6.1220\t52.80\t2.6403\t4\t304.0\t18.40\t396.90\t5.98\t22.10\n0.26169\t0.00\t9.900\t0\t0.5440\t6.0230\t90.40\t2.8340\t4\t304.0\t18.40\t396.30\t11.72\t19.40\n0.26938\t0.00\t9.900\t0\t0.5440\t6.2660\t82.80\t3.2628\t4\t304.0\t18.40\t393.39\t7.90\t21.60\n0.36920\t0.00\t9.900\t0\t0.5440\t6.5670\t87.30\t3.6023\t4\t304.0\t18.40\t395.69\t9.28\t23.80\n0.25356\t0.00\t9.900\t0\t0.5440\t5.7050\t77.70\t3.9450\t4\t304.0\t18.40\t396.42\t11.50\t16.20\n0.31827\t0.00\t9.900\t0\t0.5440\t5.9140\t83.20\t3.9986\t4\t304.0\t18.40\t390.70\t18.33\t17.80\n0.24522\t0.00\t9.900\t0\t0.5440\t5.7820\t71.70\t4.0317\t4\t304.0\t18.40\t396.90\t15.94\t19.80\n0.40202\t0.00\t9.900\t0\t0.5440\t6.3820\t67.20\t3.5325\t4\t304.0\t18.40\t395.21\t10.36\t23.10\n0.47547\t0.00\t9.900\t0\t0.5440\t6.1130\t58.80\t4.0019\t4\t304.0\t18.40\t396.23\t12.73\t21.00\n0.16760\t0.00\t7.380\t0\t0.4930\t6.4260\t52.30\t4.5404\t5\t287.0\t19.60\t396.90\t7.20\t23.80\n0.18159\t0.00\t7.380\t0\t0.4930\t6.3760\t54.30\t4.5404\t5\t287.0\t19.60\t396.90\t6.87\t23.10\n0.35114\t0.00\t7.380\t0\t0.4930\t6.0410\t49.90\t4.7211\t5\t287.0\t19.60\t396.90\t7.70\t20.40\n0.28392\t0.00\t7.380\t0\t0.4930\t5.7080\t74.30\t4.7211\t5\t287.0\t19.60\t391.13\t11.74\t18.50\n0.34109\t0.00\t7.380\t0\t0.4930\t6.4150\t40.10\t4.7211\t5\t287.0\t19.60\t396.90\t6.12\t25.00\n0.19186\t0.00\t7.380\t0\t0.4930\t6.4310\t14.70\t5.4159\t5\t287.0\t19.60\t393.68\t5.08\t24.60\n0.30347\t0.00\t7.380\t0\t0.4930\t6.3120\t28.90\t5.4159\t5\t287.0\t19.60\t396.90\t6.15\t23.00\n0.24103\t0.00\t7.380\t0\t0.4930\t6.0830\t43.70\t5.4159\t5\t287.0\t19.60\t396.90\t12.79\t22.20\n0.06617\t0.00\t3.240\t0\t0.4600\t5.8680\t25.80\t5.2146\t4\t430.0\t16.90\t382.44\t9.97\t19.30\n0.06724\t0.00\t3.240\t0\t0.4600\t6.3330\t17.20\t5.2146\t4\t430.0\t16.90\t375.21\t7.34\t22.60\n0.04544\t0.00\t3.240\t0\t0.4600\t6.1440\t32.20\t5.8736\t4\t430.0\t16.90\t368.57\t9.09\t19.80\n0.05023\t35.00\t6.060\t0\t0.4379\t5.7060\t28.40\t6.6407\t1\t304.0\t16.90\t394.02\t12.43\t17.10\n0.03466\t35.00\t6.060\t0\t0.4379\t6.0310\t23.30\t6.6407\t1\t304.0\t16.90\t362.25\t7.83\t19.40\n0.05083\t0.00\t5.190\t0\t0.5150\t6.3160\t38.10\t6.4584\t5\t224.0\t20.20\t389.71\t5.68\t22.20\n0.03738\t0.00\t5.190\t0\t0.5150\t6.3100\t38.50\t6.4584\t5\t224.0\t20.20\t389.40\t6.75\t20.70\n0.03961\t0.00\t5.190\t0\t0.5150\t6.0370\t34.50\t5.9853\t5\t224.0\t20.20\t396.90\t8.01\t21.10\n0.03427\t0.00\t5.190\t0\t0.5150\t5.8690\t46.30\t5.2311\t5\t224.0\t20.20\t396.90\t9.80\t19.50\n0.03041\t0.00\t5.190\t0\t0.5150\t5.8950\t59.60\t5.6150\t5\t224.0\t20.20\t394.81\t10.56\t18.50\n0.03306\t0.00\t5.190\t0\t0.5150\t6.0590\t37.30\t4.8122\t5\t224.0\t20.20\t396.14\t8.51\t20.60\n0.05497\t0.00\t5.190\t0\t0.5150\t5.9850\t45.40\t4.8122\t5\t224.0\t20.20\t396.90\t9.74\t19.00\n0.06151\t0.00\t5.190\t0\t0.5150\t5.9680\t58.50\t4.8122\t5\t224.0\t20.20\t396.90\t9.29\t18.70\n0.01301\t35.00\t1.520\t0\t0.4420\t7.2410\t49.30\t7.0379\t1\t284.0\t15.50\t394.74\t5.49\t32.70\n0.02498\t0.00\t1.890\t0\t0.5180\t6.5400\t59.70\t6.2669\t1\t422.0\t15.90\t389.96\t8.65\t16.50\n0.02543\t55.00\t3.780\t0\t0.4840\t6.6960\t56.40\t5.7321\t5\t370.0\t17.60\t396.90\t7.18\t23.90\n0.03049\t55.00\t3.780\t0\t0.4840\t6.8740\t28.10\t6.4654\t5\t370.0\t17.60\t387.97\t4.61\t31.20\n0.03113\t0.00\t4.390\t0\t0.4420\t6.0140\t48.50\t8.0136\t3\t352.0\t18.80\t385.64\t10.53\t17.50\n0.06162\t0.00\t4.390\t0\t0.4420\t5.8980\t52.30\t8.0136\t3\t352.0\t18.80\t364.61\t12.67\t17.20\n0.01870\t85.00\t4.150\t0\t0.4290\t6.5160\t27.70\t8.5353\t4\t351.0\t17.90\t392.43\t6.36\t23.10\n0.01501\t80.00\t2.010\t0\t0.4350\t6.6350\t29.70\t8.3440\t4\t280.0\t17.00\t390.94\t5.99\t24.50\n0.02899\t40.00\t1.250\t0\t0.4290\t6.9390\t34.50\t8.7921\t1\t335.0\t19.70\t389.85\t5.89\t26.60\n0.06211\t40.00\t1.250\t0\t0.4290\t6.4900\t44.40\t8.7921\t1\t335.0\t19.70\t396.90\t5.98\t22.90\n0.07950\t60.00\t1.690\t0\t0.4110\t6.5790\t35.90\t10.7103\t4\t411.0\t18.30\t370.78\t5.49\t24.10\n0.07244\t60.00\t1.690\t0\t0.4110\t5.8840\t18.50\t10.7103\t4\t411.0\t18.30\t392.33\t7.79\t18.60\n0.01709\t90.00\t2.020\t0\t0.4100\t6.7280\t36.10\t12.1265\t5\t187.0\t17.00\t384.46\t4.50\t30.10\n0.04301\t80.00\t1.910\t0\t0.4130\t5.6630\t21.90\t10.5857\t4\t334.0\t22.00\t382.80\t8.05\t18.20\n0.10659\t80.00\t1.910\t0\t0.4130\t5.9360\t19.50\t10.5857\t4\t334.0\t22.00\t376.04\t5.57\t20.60\n8.98296\t0.00\t18.100\t1\t0.7700\t6.2120\t97.40\t2.1222\t24\t666.0\t20.20\t377.73\t17.60\t17.80\n3.84970\t0.00\t18.100\t1\t0.7700\t6.3950\t91.00\t2.5052\t24\t666.0\t20.20\t391.34\t13.27\t21.70\n5.20177\t0.00\t18.100\t1\t0.7700\t6.1270\t83.40\t2.7227\t24\t666.0\t20.20\t395.43\t11.48\t22.70\n4.26131\t0.00\t18.100\t0\t0.7700\t6.1120\t81.30\t2.5091\t24\t666.0\t20.20\t390.74\t12.67\t22.60\n4.54192\t0.00\t18.100\t0\t0.7700\t6.3980\t88.00\t2.5182\t24\t666.0\t20.20\t374.56\t7.79\t25.00\n3.83684\t0.00\t18.100\t0\t0.7700\t6.2510\t91.10\t2.2955\t24\t666.0\t20.20\t350.65\t14.19\t19.90\n3.67822\t0.00\t18.100\t0\t0.7700\t5.3620\t96.20\t2.1036\t24\t666.0\t20.20\t380.79\t10.19\t20.80\n4.22239\t0.00\t18.100\t1\t0.7700\t5.8030\t89.00\t1.9047\t24\t666.0\t20.20\t353.04\t14.64\t16.80\n3.47428\t0.00\t18.100\t1\t0.7180\t8.7800\t82.90\t1.9047\t24\t666.0\t20.20\t354.55\t5.29\t21.90\n4.55587\t0.00\t18.100\t0\t0.7180\t3.5610\t87.90\t1.6132\t24\t666.0\t20.20\t354.70\t7.12\t27.50\n3.69695\t0.00\t18.100\t0\t0.7180\t4.9630\t91.40\t1.7523\t24\t666.0\t20.20\t316.03\t14.00\t21.90\n3.52220\t0.00\t18.100\t0\t0.6310\t3.8630\t100.00\t1.5106\t24\t666.0\t20.20\t131.42\t13.33\t23.10\n4.89822\t0.00\t18.100\t0\t0.6310\t4.9700\t100.00\t1.3325\t24\t666.0\t20.20\t375.52\t3.26\t50.00\n5.66998\t0.00\t18.100\t1\t0.6310\t6.6830\t96.80\t1.3567\t24\t666.0\t20.20\t375.33\t3.73\t50.00\n6.53876\t0.00\t18.100\t1\t0.6310\t7.0160\t97.50\t1.2024\t24\t666.0\t20.20\t392.05\t2.96\t50.00\n9.23230\t0.00\t18.100\t0\t0.6310\t6.2160\t100.00\t1.1691\t24\t666.0\t20.20\t366.15\t9.53\t50.00\n8.26725\t0.00\t18.100\t1\t0.6680\t5.8750\t89.60\t1.1296\t24\t666.0\t20.20\t347.88\t8.88\t50.00\n1.10810\t0.00\t18.100\t0\t0.6680\t4.9060\t100.00\t1.1742\t24\t666.0\t20.20\t396.90\t34.77\t13.80\n8.49820\t0.00\t18.100\t0\t0.6680\t4.1380\t100.00\t1.1370\t24\t666.0\t20.20\t396.90\t37.97\t13.80\n9.60910\t0.00\t18.100\t0\t0.6710\t7.3130\t97.90\t1.3163\t24\t666.0\t20.20\t396.90\t13.44\t15.00\n5.28800\t0.00\t18.100\t0\t0.6710\t6.6490\t93.30\t1.3449\t24\t666.0\t20.20\t363.02\t23.24\t13.90\n9.82349\t0.00\t18.100\t0\t0.6710\t6.7940\t98.80\t1.3580\t24\t666.0\t20.20\t396.90\t21.24\t13.30\n3.64820\t0.00\t18.100\t0\t0.6710\t6.3800\t96.20\t1.3861\t24\t666.0\t20.20\t396.90\t23.69\t13.10\n7.86670\t0.00\t18.100\t0\t0.6710\t6.2230\t100.00\t1.3861\t24\t666.0\t20.20\t393.74\t21.78\t10.20\n8.97620\t0.00\t18.100\t0\t0.6710\t6.9680\t91.90\t1.4165\t24\t666.0\t20.20\t396.90\t17.21\t10.40\n5.87440\t0.00\t18.100\t0\t0.6710\t6.5450\t99.10\t1.5192\t24\t666.0\t20.20\t396.90\t21.08\t10.90\n9.18702\t0.00\t18.100\t0\t0.7000\t5.5360\t100.00\t1.5804\t24\t666.0\t20.20\t396.90\t23.60\t11.30\n7.99248\t0.00\t18.100\t0\t0.7000\t5.5200\t100.00\t1.5331\t24\t666.0\t20.20\t396.90\t24.56\t12.30\n0.08490\t0.00\t18.100\t0\t0.7000\t4.3680\t91.20\t1.4395\t24\t666.0\t20.20\t285.83\t30.63\t8.80\n6.81180\t0.00\t18.100\t0\t0.7000\t5.2770\t98.10\t1.4261\t24\t666.0\t20.20\t396.90\t30.81\t7.20\n4.39380\t0.00\t18.100\t0\t0.7000\t4.6520\t100.00\t1.4672\t24\t666.0\t20.20\t396.90\t28.28\t10.50\n2.59710\t0.00\t18.100\t0\t0.7000\t5.0000\t89.50\t1.5184\t24\t666.0\t20.20\t396.90\t31.99\t7.40\n4.33370\t0.00\t18.100\t0\t0.7000\t4.8800\t100.00\t1.5895\t24\t666.0\t20.20\t372.92\t30.62\t10.20\n8.15174\t0.00\t18.100\t0\t0.7000\t5.3900\t98.90\t1.7281\t24\t666.0\t20.20\t396.90\t20.85\t11.50\n6.96215\t0.00\t18.100\t0\t0.7000\t5.7130\t97.00\t1.9265\t24\t666.0\t20.20\t394.43\t17.11\t15.10\n5.29305\t0.00\t18.100\t0\t0.7000\t6.0510\t82.50\t2.1678\t24\t666.0\t20.20\t378.38\t18.76\t23.20\n1.57790\t0.00\t18.100\t0\t0.7000\t5.0360\t97.00\t1.7700\t24\t666.0\t20.20\t396.90\t25.68\t9.70\n8.64476\t0.00\t18.100\t0\t0.6930\t6.1930\t92.60\t1.7912\t24\t666.0\t20.20\t396.90\t15.17\t13.80\n3.35980\t0.00\t18.100\t0\t0.6930\t5.8870\t94.70\t1.7821\t24\t666.0\t20.20\t396.90\t16.35\t12.70\n8.71675\t0.00\t18.100\t0\t0.6930\t6.4710\t98.80\t1.7257\t24\t666.0\t20.20\t391.98\t17.12\t13.10\n5.87205\t0.00\t18.100\t0\t0.6930\t6.4050\t96.00\t1.6768\t24\t666.0\t20.20\t396.90\t19.37\t12.50\n7.67202\t0.00\t18.100\t0\t0.6930\t5.7470\t98.90\t1.6334\t24\t666.0\t20.20\t393.10\t19.92\t8.50\n8.35180\t0.00\t18.100\t0\t0.6930\t5.4530\t100.00\t1.4896\t24\t666.0\t20.20\t396.90\t30.59\t5.00\n9.91655\t0.00\t18.100\t0\t0.6930\t5.8520\t77.80\t1.5004\t24\t666.0\t20.20\t338.16\t29.97\t6.30\n5.04610\t0.00\t18.100\t0\t0.6930\t5.9870\t100.00\t1.5888\t24\t666.0\t20.20\t396.90\t26.77\t5.60\n4.23620\t0.00\t18.100\t0\t0.6930\t6.3430\t100.00\t1.5741\t24\t666.0\t20.20\t396.90\t20.32\t7.20\n9.59571\t0.00\t18.100\t0\t0.6930\t6.4040\t100.00\t1.6390\t24\t666.0\t20.20\t376.11\t20.31\t12.10\n4.80170\t0.00\t18.100\t0\t0.6930\t5.3490\t96.00\t1.7028\t24\t666.0\t20.20\t396.90\t19.77\t8.30\n1.52920\t0.00\t18.100\t0\t0.6930\t5.5310\t85.40\t1.6074\t24\t666.0\t20.20\t329.46\t27.38\t8.50\n7.92080\t0.00\t18.100\t0\t0.6930\t5.6830\t100.00\t1.4254\t24\t666.0\t20.20\t384.97\t22.98\t5.00\n0.71620\t0.00\t18.100\t0\t0.6590\t4.1380\t100.00\t1.1781\t24\t666.0\t20.20\t370.22\t23.34\t11.90\n1.95110\t0.00\t18.100\t0\t0.6590\t5.6080\t100.00\t1.2852\t24\t666.0\t20.20\t332.09\t12.13\t27.90\n7.40389\t0.00\t18.100\t0\t0.5970\t5.6170\t97.90\t1.4547\t24\t666.0\t20.20\t314.64\t26.40\t17.20\n4.43830\t0.00\t18.100\t0\t0.5970\t6.8520\t100.00\t1.4655\t24\t666.0\t20.20\t179.36\t19.78\t27.50\n1.13580\t0.00\t18.100\t0\t0.5970\t5.7570\t100.00\t1.4130\t24\t666.0\t20.20\t2.60\t10.11\t15.00\n4.05070\t0.00\t18.100\t0\t0.5970\t6.6570\t100.00\t1.5275\t24\t666.0\t20.20\t35.05\t21.22\t17.20\n8.81100\t0.00\t18.100\t0\t0.5970\t4.6280\t100.00\t1.5539\t24\t666.0\t20.20\t28.79\t34.37\t17.90\n8.65580\t0.00\t18.100\t0\t0.5970\t5.1550\t100.00\t1.5894\t24\t666.0\t20.20\t210.97\t20.08\t16.30\n5.74610\t0.00\t18.100\t0\t0.6930\t4.5190\t100.00\t1.6582\t24\t666.0\t20.20\t88.27\t36.98\t7.00\n8.08460\t0.00\t18.100\t0\t0.6790\t6.4340\t100.00\t1.8347\t24\t666.0\t20.20\t27.25\t29.05\t7.20\n0.83420\t0.00\t18.100\t0\t0.6790\t6.7820\t90.80\t1.8195\t24\t666.0\t20.20\t21.57\t25.79\t7.50\n5.94060\t0.00\t18.100\t0\t0.6790\t5.3040\t89.10\t1.6475\t24\t666.0\t20.20\t127.36\t26.64\t10.40\n3.53410\t0.00\t18.100\t0\t0.6790\t5.9570\t100.00\t1.8026\t24\t666.0\t20.20\t16.45\t20.62\t8.80\n1.81230\t0.00\t18.100\t0\t0.7180\t6.8240\t76.50\t1.7940\t24\t666.0\t20.20\t48.45\t22.74\t8.40\n1.08740\t0.00\t18.100\t0\t0.7180\t6.4110\t100.00\t1.8589\t24\t666.0\t20.20\t318.75\t15.02\t16.70\n7.02259\t0.00\t18.100\t0\t0.7180\t6.0060\t95.30\t1.8746\t24\t666.0\t20.20\t319.98\t15.70\t14.20\n2.04820\t0.00\t18.100\t0\t0.6140\t5.6480\t87.60\t1.9512\t24\t666.0\t20.20\t291.55\t14.10\t20.80\n7.05042\t0.00\t18.100\t0\t0.6140\t6.1030\t85.10\t2.0218\t24\t666.0\t20.20\t2.52\t23.29\t13.40\n8.79212\t0.00\t18.100\t0\t0.5840\t5.5650\t70.60\t2.0635\t24\t666.0\t20.20\t3.65\t17.16\t11.70\n5.86030\t0.00\t18.100\t0\t0.6790\t5.8960\t95.40\t1.9096\t24\t666.0\t20.20\t7.68\t24.39\t8.30\n2.24720\t0.00\t18.100\t0\t0.5840\t5.8370\t59.70\t1.9976\t24\t666.0\t20.20\t24.65\t15.69\t10.20\n7.66190\t0.00\t18.100\t0\t0.6790\t6.2020\t78.70\t1.8629\t24\t666.0\t20.20\t18.82\t14.52\t10.90\n7.36711\t0.00\t18.100\t0\t0.6790\t6.1930\t78.10\t1.9356\t24\t666.0\t20.20\t96.73\t21.52\t11.00\n9.33889\t0.00\t18.100\t0\t0.6790\t6.3800\t95.60\t1.9682\t24\t666.0\t20.20\t60.72\t24.08\t9.50\n8.49213\t0.00\t18.100\t0\t0.5840\t6.3480\t86.10\t2.0527\t24\t666.0\t20.20\t83.45\t17.64\t14.50\n0.06230\t0.00\t18.100\t0\t0.5840\t6.8330\t94.30\t2.0882\t24\t666.0\t20.20\t81.33\t19.69\t14.10\n6.44405\t0.00\t18.100\t0\t0.5840\t6.4250\t74.80\t2.2004\t24\t666.0\t20.20\t97.95\t12.03\t16.10\n5.58107\t0.00\t18.100\t0\t0.7130\t6.4360\t87.90\t2.3158\t24\t666.0\t20.20\t100.19\t16.22\t14.30\n3.91340\t0.00\t18.100\t0\t0.7130\t6.2080\t95.00\t2.2222\t24\t666.0\t20.20\t100.63\t15.17\t11.70\n1.16040\t0.00\t18.100\t0\t0.7400\t6.6290\t94.60\t2.1247\t24\t666.0\t20.20\t109.85\t23.27\t13.40\n4.42080\t0.00\t18.100\t0\t0.7400\t6.4610\t93.30\t2.0026\t24\t666.0\t20.20\t27.49\t18.05\t9.60\n5.17720\t0.00\t18.100\t0\t0.7400\t6.1520\t100.00\t1.9142\t24\t666.0\t20.20\t9.32\t26.45\t8.70\n3.67810\t0.00\t18.100\t0\t0.7400\t5.9350\t87.90\t1.8206\t24\t666.0\t20.20\t68.95\t34.02\t8.40\n9.39063\t0.00\t18.100\t0\t0.7400\t5.6270\t93.90\t1.8172\t24\t666.0\t20.20\t396.90\t22.88\t12.80\n2.05110\t0.00\t18.100\t0\t0.7400\t5.8180\t92.40\t1.8662\t24\t666.0\t20.20\t391.45\t22.11\t10.50\n9.72418\t0.00\t18.100\t0\t0.7400\t6.4060\t97.20\t2.0651\t24\t666.0\t20.20\t385.96\t19.52\t17.10\n5.66637\t0.00\t18.100\t0\t0.7400\t6.2190\t100.00\t2.0048\t24\t666.0\t20.20\t395.69\t16.59\t18.40\n9.96654\t0.00\t18.100\t0\t0.7400\t6.4850\t100.00\t1.9784\t24\t666.0\t20.20\t386.73\t18.85\t15.40\n2.80230\t0.00\t18.100\t0\t0.7400\t5.8540\t96.60\t1.8956\t24\t666.0\t20.20\t240.52\t23.79\t10.80\n0.67180\t0.00\t18.100\t0\t0.7400\t6.4590\t94.80\t1.9879\t24\t666.0\t20.20\t43.06\t23.98\t11.80\n6.28807\t0.00\t18.100\t0\t0.7400\t6.3410\t96.40\t2.0720\t24\t666.0\t20.20\t318.01\t17.79\t14.90\n9.92485\t0.00\t18.100\t0\t0.7400\t6.2510\t96.60\t2.1980\t24\t666.0\t20.20\t388.52\t16.44\t12.60\n9.32909\t0.00\t18.100\t0\t0.7130\t6.1850\t98.70\t2.2616\t24\t666.0\t20.20\t396.90\t18.13\t14.10\n7.52601\t0.00\t18.100\t0\t0.7130\t6.4170\t98.30\t2.1850\t24\t666.0\t20.20\t304.21\t19.31\t13.00\n6.71772\t0.00\t18.100\t0\t0.7130\t6.7490\t92.60\t2.3236\t24\t666.0\t20.20\t0.32\t17.44\t13.40\n5.44114\t0.00\t18.100\t0\t0.7130\t6.6550\t98.20\t2.3552\t24\t666.0\t20.20\t355.29\t17.73\t15.20\n5.09017\t0.00\t18.100\t0\t0.7130\t6.2970\t91.80\t2.3682\t24\t666.0\t20.20\t385.09\t17.27\t16.10\n8.24809\t0.00\t18.100\t0\t0.7130\t7.3930\t99.30\t2.4527\t24\t666.0\t20.20\t375.87\t16.74\t17.80\n9.51363\t0.00\t18.100\t0\t0.7130\t6.7280\t94.10\t2.4961\t24\t666.0\t20.20\t6.68\t18.71\t14.90\n4.75237\t0.00\t18.100\t0\t0.7130\t6.5250\t86.50\t2.4358\t24\t666.0\t20.20\t50.92\t18.13\t14.10\n4.66883\t0.00\t18.100\t0\t0.7130\t5.9760\t87.90\t2.5806\t24\t666.0\t20.20\t10.48\t19.01\t12.70\n8.20058\t0.00\t18.100\t0\t0.7130\t5.9360\t80.30\t2.7792\t24\t666.0\t20.20\t3.50\t16.94\t13.50\n7.75223\t0.00\t18.100\t0\t0.7130\t6.3010\t83.70\t2.7831\t24\t666.0\t20.20\t272.21\t16.23\t14.90\n6.80117\t0.00\t18.100\t0\t0.7130\t6.0810\t84.40\t2.7175\t24\t666.0\t20.20\t396.90\t14.70\t20.00\n4.81213\t0.00\t18.100\t0\t0.7130\t6.7010\t90.00\t2.5975\t24\t666.0\t20.20\t255.23\t16.42\t16.40\n3.69311\t0.00\t18.100\t0\t0.7130\t6.3760\t88.40\t2.5671\t24\t666.0\t20.20\t391.43\t14.65\t17.70\n6.65492\t0.00\t18.100\t0\t0.7130\t6.3170\t83.00\t2.7344\t24\t666.0\t20.20\t396.90\t13.99\t19.50\n5.82115\t0.00\t18.100\t0\t0.7130\t6.5130\t89.90\t2.8016\t24\t666.0\t20.20\t393.82\t10.29\t20.20\n7.83932\t0.00\t18.100\t0\t0.6550\t6.2090\t65.40\t2.9634\t24\t666.0\t20.20\t396.90\t13.22\t21.40\n3.16360\t0.00\t18.100\t0\t0.6550\t5.7590\t48.20\t3.0665\t24\t666.0\t20.20\t334.40\t14.13\t19.90\n3.77498\t0.00\t18.100\t0\t0.6550\t5.9520\t84.70\t2.8715\t24\t666.0\t20.20\t22.01\t17.15\t19.00\n4.42228\t0.00\t18.100\t0\t0.5840\t6.0030\t94.50\t2.5403\t24\t666.0\t20.20\t331.29\t21.32\t19.10\n5.57570\t0.00\t18.100\t0\t0.5800\t5.9260\t71.00\t2.9084\t24\t666.0\t20.20\t368.74\t18.13\t19.10\n3.07510\t0.00\t18.100\t0\t0.5800\t5.7130\t56.70\t2.8237\t24\t666.0\t20.20\t396.90\t14.76\t20.10\n4.34879\t0.00\t18.100\t0\t0.5800\t6.1670\t84.00\t3.0334\t24\t666.0\t20.20\t396.90\t16.29\t19.90\n4.03841\t0.00\t18.100\t0\t0.5320\t6.2290\t90.70\t3.0993\t24\t666.0\t20.20\t395.33\t12.87\t19.60\n3.56868\t0.00\t18.100\t0\t0.5800\t6.4370\t75.00\t2.8965\t24\t666.0\t20.20\t393.37\t14.36\t23.20\n4.64689\t0.00\t18.100\t0\t0.6140\t6.9800\t67.60\t2.5329\t24\t666.0\t20.20\t374.68\t11.66\t29.80\n8.05579\t0.00\t18.100\t0\t0.5840\t5.4270\t95.40\t2.4298\t24\t666.0\t20.20\t352.58\t18.14\t13.80\n6.39312\t0.00\t18.100\t0\t0.5840\t6.1620\t97.40\t2.2060\t24\t666.0\t20.20\t302.76\t24.10\t13.30\n4.87141\t0.00\t18.100\t0\t0.6140\t6.4840\t93.60\t2.3053\t24\t666.0\t20.20\t396.21\t18.68\t16.70\n5.02340\t0.00\t18.100\t0\t0.6140\t5.3040\t97.30\t2.1007\t24\t666.0\t20.20\t349.48\t24.91\t12.00\n0.23300\t0.00\t18.100\t0\t0.6140\t6.1850\t96.70\t2.1705\t24\t666.0\t20.20\t379.70\t18.03\t14.60\n4.33370\t0.00\t18.100\t0\t0.6140\t6.2290\t88.00\t1.9512\t24\t666.0\t20.20\t383.32\t13.11\t21.40\n5.82401\t0.00\t18.100\t0\t0.5320\t6.2420\t64.70\t3.4242\t24\t666.0\t20.20\t396.90\t10.74\t23.00\n5.70818\t0.00\t18.100\t0\t0.5320\t6.7500\t74.90\t3.3317\t24\t666.0\t20.20\t393.07\t7.74\t23.70\n5.73116\t0.00\t18.100\t0\t0.5320\t7.0610\t77.00\t3.4106\t24\t666.0\t20.20\t395.28\t7.01\t25.00\n2.81838\t0.00\t18.100\t0\t0.5320\t5.7620\t40.30\t4.0983\t24\t666.0\t20.20\t392.92\t10.42\t21.80\n2.37857\t0.00\t18.100\t0\t0.5830\t5.8710\t41.90\t3.7240\t24\t666.0\t20.20\t370.73\t13.34\t20.60\n3.67367\t0.00\t18.100\t0\t0.5830\t6.3120\t51.90\t3.9917\t24\t666.0\t20.20\t388.62\t10.58\t21.20\n5.69175\t0.00\t18.100\t0\t0.5830\t6.1140\t79.80\t3.5459\t24\t666.0\t20.20\t392.68\t14.98\t19.10\n4.83567\t0.00\t18.100\t0\t0.5830\t5.9050\t53.20\t3.1523\t24\t666.0\t20.20\t388.22\t11.45\t20.60\n0.15086\t0.00\t27.740\t0\t0.6090\t5.4540\t92.70\t1.8209\t4\t711.0\t20.10\t395.09\t18.06\t15.20\n0.18337\t0.00\t27.740\t0\t0.6090\t5.4140\t98.30\t1.7554\t4\t711.0\t20.10\t344.05\t23.97\t7.00\n0.20746\t0.00\t27.740\t0\t0.6090\t5.0930\t98.00\t1.8226\t4\t711.0\t20.10\t318.43\t29.68\t8.10\n0.10574\t0.00\t27.740\t0\t0.6090\t5.9830\t98.80\t1.8681\t4\t711.0\t20.10\t390.11\t18.07\t13.60\n0.11132\t0.00\t27.740\t0\t0.6090\t5.9830\t83.50\t2.1099\t4\t711.0\t20.10\t396.90\t13.35\t20.10\n0.17331\t0.00\t9.690\t0\t0.5850\t5.7070\t54.00\t2.3817\t6\t391.0\t19.20\t396.90\t12.01\t21.80\n0.27957\t0.00\t9.690\t0\t0.5850\t5.9260\t42.60\t2.3817\t6\t391.0\t19.20\t396.90\t13.59\t24.50\n0.17899\t0.00\t9.690\t0\t0.5850\t5.6700\t28.80\t2.7986\t6\t391.0\t19.20\t393.29\t17.60\t23.10\n0.28960\t0.00\t9.690\t0\t0.5850\t5.3900\t72.90\t2.7986\t6\t391.0\t19.20\t396.90\t21.14\t19.70\n0.26838\t0.00\t9.690\t0\t0.5850\t5.7940\t70.60\t2.8927\t6\t391.0\t19.20\t396.90\t14.10\t18.30\n0.23912\t0.00\t9.690\t0\t0.5850\t6.0190\t65.30\t2.4091\t6\t391.0\t19.20\t396.90\t12.92\t21.20\n0.17783\t0.00\t9.690\t0\t0.5850\t5.5690\t73.50\t2.3999\t6\t391.0\t19.20\t395.77\t15.10\t17.50\n0.22438\t0.00\t9.690\t0\t0.5850\t6.0270\t79.70\t2.4982\t6\t391.0\t19.20\t396.90\t14.33\t16.80\n0.06263\t0.00\t11.930\t0\t0.5730\t6.5930\t69.10\t2.4786\t1\t273.0\t21.00\t391.99\t9.67\t22.40\n0.04527\t0.00\t11.930\t0\t0.5730\t6.1200\t76.70\t2.2875\t1\t273.0\t21.00\t396.90\t9.08\t20.60\n0.06076\t0.00\t11.930\t0\t0.5730\t6.9760\t91.00\t2.1675\t1\t273.0\t21.00\t396.90\t5.64\t23.90\n0.10959\t0.00\t11.930\t0\t0.5730\t6.7940\t89.30\t2.3889\t1\t273.0\t21.00\t393.45\t6.48\t22.00\n0.04741\t0.00\t11.930\t0\t0.5730\t6.0300\t80.80\t2.5050\t1\t273.0\t21.00\t396.90\t7.88\t11.90" }, { "path": "docs/input/index.fsx", "content": "(*** hide ***)\n#r \"../../src/Hype/bin/Debug/DiffSharp.dll\"\n#r \"../../src/Hype/bin/Debug/Hype.dll\"\nopen DiffSharp.AD.Float32\n\n(**\nHype: Compositional Machine Learning and Hyperparameter Optimization\n====================================================================\n\nHype is a proof-of-concept deep learning library, where you can perform optimization on [compositional](http://mathworld.wolfram.com/Composition.html) machine learning systems of many components, even when such components themselves internally perform optimization. \n\nThis is enabled by nested automatic differentiation (AD) giving you access to the automatic exact derivative of any floating-point value in your code with respect to any other. Underlying computations are run by a BLAS/LAPACK backend (OpenBLAS by default).\n\n### Automatic derivatives\n\nYou do not need to worry about supplying gradients (or Hessians) of your models, which are computed exactly and efficiently by AD. The underlying AD functionality is provided by [DiffSharp](http://diffsharp.github.io/DiffSharp/index.html). \n\n\"Reverse mode\" AD is a generalized form of \"backpropagation\" and is distinct from numerical or symbolic differentiation.\n\nIn addition to reverse AD, Hype makes use of forward AD and nested combinations of forward and reverse AD. The core [differentiation API](http://diffsharp.github.io/DiffSharp/api-overview.html) provides gradients, Hessians, Jacobians, directional derivatives, and matrix-free exact Hessian- and Jacobian-vector products.\n\n### Hypergradients\n\nYou can get exact gradients of the training or validation loss with respect to hyperparameters. These __hypergradients__ allow you to do gradient-based optimization of gradient-based optimization, meaning that you can do things like optimizing learning rate and momentum schedules, weight initialization parameters, or step sizes and mass matrices in Hamiltonian Monte Carlo models. (A recent article doing this with Python: _Maclaurin, Dougal, David Duvenaud, and Ryan P. Adams. \"Gradient-based Hyperparameter Optimization through Reversible Learning.\" arXiv preprint arXiv:1502.03492 (2015)._)\n\n*)\n\nopen Hype\nopen Hype.Neural\n\n// Train a network with stochastic gradient descent and a learning rate schedule\nlet train (x:DV) = \n let n = FeedForward()\n n.Add(Linear(784, 300))\n n.Add(tanh)\n n.Add(Linear(300, 10))\n let loss, _ = Layer.Train(n, data, {Params.Default with \n LearningRate = Schedule x\n Momentum = Momentum.DefaultNesterov\n Batch = Minibatch 100\n Loss = CrossEntropyOnLinear})\n loss // Return the loss at the end of training\n\n// Train the training, i.e., optimize the learning schedule vector by using its hypergradient\nlet hypertrain = \n Optimize.Minimize(train, DV.create 200 (D 1.f), {Params.Default with Epochs = 50})\n\n(**\n\nYou can also take derivatives with respect to training data, to analyze training sensitivities.\n\n### Compositionality\n\nNested AD handles higher-order derivatives up to any level, including in complex cases such as \n\n$$$\n \\mathbf{min} \\left(x \\; \\mapsto \\; f(x) + \\mathbf{min} \\left( y \\; \\mapsto \\; g(x,\\,y) \\right) \\right)\\, ,\n\nwhere $\\mathbf{min}$ uses gradient-based optimization. (Note that the inner function has a reference to the argument of the outer function.) This allows you to create complex systems where many components may internally perform optimization.\n\nFor example, you can optimize the rules of a multi-player game where the players themselves optimize their own strategy using a simple model of the opponent which they optimize according to their opponent's observed behaviour. \n\nOr you can perform optimization of procedures that are internally using differentiation for purposes other than optimization, such as adaptive control or simulations.\n\n### Complex objective functions\n\nYou can use derivatives in the definition of objective functions for training your models. For example, your objective function can take input sensitivities into account, for training models that are invariant under a set of input transformations.\n\nRoadmap\n-------\n\n
| @Model.TableHeader | Description |
\n @{ var id = Html.UniqueID().ToString(); }\n \n @Html.Encode(it.Details.FormatUsage(40))\n \n \n Signature: @Html.Encode(it.Details.Signature) \n \n @if (!it.Details.Modifiers.IsEmpty) {\n Modifiers: @it.Details.FormatModifiers \n }\n @if (!it.Details.TypeArguments.IsEmpty) {\n Type parameters: @it.Details.FormatTypeArguments\n }\n | \n \n @if (!String.IsNullOrEmpty(it.Details.FormatSourceLocation))\n {\n \n ![](\"../content/img/github.png\") \n ![](\"../content/img/github-blue.png\") \n \n }\n @it.Comment.FullText\n | \n
| Type | Description |
| \n @it.Name\n | \n@it.Comment.Blurb | \n
| Module | Description |
| \n @it.Name\n | \n@it.Comment.Blurb | \n
![](\"../img/hype.png\")
![](\"img/hype.png\")