Repository: saurfang/spark-tsne
Branch: master
Commit: ba691f5a4857
Files: 30
Total size: 57.0 KB
Directory structure:
gitextract_xem_b0op/
├── .gitattributes
├── .gitignore
├── .travis.yml
├── LICENSE
├── README.md
├── build.sbt
├── data/
│ └── mnist/
│ └── tsne.R
├── project/
│ ├── Common.scala
│ ├── Dependencies.scala
│ ├── SparkSubmit.scala
│ ├── build.properties
│ └── plugins.sbt
├── spark-tsne-core/
│ └── src/
│ ├── main/
│ │ └── scala/
│ │ ├── com/
│ │ │ └── github/
│ │ │ └── saurfang/
│ │ │ └── spark/
│ │ │ └── tsne/
│ │ │ ├── TSNEGradient.scala
│ │ │ ├── TSNEHelper.scala
│ │ │ ├── TSNEParam.scala
│ │ │ ├── X2P.scala
│ │ │ ├── impl/
│ │ │ │ ├── BHTSNE.scala
│ │ │ │ ├── LBFGSTSNE.scala
│ │ │ │ └── SimpleTSNE.scala
│ │ │ └── tree/
│ │ │ └── SPTree.scala
│ │ └── org/
│ │ └── apache/
│ │ └── spark/
│ │ └── mllib/
│ │ └── X2PHelper.scala
│ └── test/
│ └── scala/
│ ├── com/
│ │ └── github/
│ │ └── saurfang/
│ │ └── spark/
│ │ └── tsne/
│ │ ├── BugDemonstrationTest.scala
│ │ ├── TSNEGradientTest.scala
│ │ ├── X2PSuite.scala
│ │ └── tree/
│ │ └── SPTreeSpec.scala
│ └── org/
│ └── apache/
│ └── spark/
│ ├── LocalSparkContext.scala
│ └── SharedSparkContext.scala
├── spark-tsne-examples/
│ └── src/
│ └── main/
│ ├── resources/
│ │ └── log4j.properties
│ └── scala/
│ └── com/
│ └── github/
│ └── saurfang/
│ └── spark/
│ └── tsne/
│ └── examples/
│ └── MNIST.scala
└── spark-tsne-player/
└── src/
└── main/
└── html/
└── tsne.html
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitattributes
================================================
*.gz filter=lfs diff=lfs merge=lfs -text
*.gif filter=lfs diff=lfs merge=lfs -text
*.json filter=lfs diff=lfs merge=lfs -text
================================================
FILE: .gitignore
================================================
/RUNNING_PID
/logs/
project/project/
project/target/
target/
.idea
.tmp
================================================
FILE: .travis.yml
================================================
language: scala
scala:
- 2.10.6
- 2.11.7
jdk:
- oraclejdk8
- oraclejdk7
- openjdk7
================================================
FILE: LICENSE
================================================
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "{}"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2015 Forest Fang
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
===================================
This project also contains code from TSne.jl and d3.js.
License can be found at:
https://github.com/lejon/TSne.jl/blob/master/LICENSE.md
and
https://github.com/mbostock/d3/blob/master/LICENSE
respectively.
================================================
FILE: README.md
================================================
# spark-tsne
[](https://gitter.im/saurfang/spark-tsne?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) [](https://travis-ci.org/erwinvaneijk/spark-tsne)
Distributed [t-SNE](http://lvdmaaten.github.io/tsne/) with Apache Spark. WIP...
t-SNE is a dimension reduction technique that is particularly good for visualizing high
dimensional data. This is an attempt to implement this algorithm using Spark to leverage
distributed computing power.
The project is still in progress of replicating reference implementations from the original
papers. Spark specific optimizations will be the next goal once the correctness is verified.
Currently I'm showcasing this using the standard [MNIST](http://yann.lecun.com/exdb/mnist/)
handwriting recognition dataset. I have created a [WebGL player](https://saurfang.github.io/spark-tsne-demo/tsne-pixi.html)
(built using [pixi.js](https://github.com/pixijs/pixi.js)) to visualize the inner workings
as well as the final results of t-SNE. If a WebGL is unavailable for you, you may checkout
the [d3.js player](https://saurfang.github.io/spark-tsne-demo/tsne.html) instead.
## Credits
- [t-SNE Julia implementation](https://github.com/lejon/TSne.jl)
- [Barnes-Hut t-SNE](https://github.com/lvdmaaten/bhtsne/)
================================================
FILE: build.sbt
================================================
import Common._
lazy val root = Project("spark-tsne", file(".")).
settings(commonSettings: _*).
aggregate(core, vis, examples)
lazy val core = tsneProject("spark-tsne-core").
settings(Dependencies.core)
lazy val vis = tsneProject("spark-tsne-player").
dependsOn(core)
lazy val examples = tsneProject("spark-tsne-examples").
dependsOn(core, vis).
settings(fork in run := true).
settings(Dependencies.core).
settings(SparkSubmit.settings: _*)
================================================
FILE: data/mnist/tsne.R
================================================
library(dplyr)
library(ggplot2)
library(animation)
library(jsonlite)
resultFiles <- list.files("~/GitHub/spark-tsne/.tmp/MNIST/", "result", full.names = TRUE)
results <- lapply(resultFiles, function(file) { read.csv(file, FALSE) })
resultsCombined <- lapply(1:length(results), function(i) {
result <- results[[i]]
names(result) <- c("label", "x", "y")
mutate(result, i = i, key = row_number())
}) %>%
rbind_all()
#### save results as json for viewer ####
iterations <- c(1:99, seq(100, length(results), 5)) # assume 100 early exaggeration here
resultsByObs <- filter(resultsCombined, i %in% iterations) %>%
group_by(key) %>%
# do({
# list(key = unbox(.$key[1]), label = unbox(.$label[1]),
# # assume order will preserve
# pos = select(., x, y)) %>%
# data_frame
# })
do(key = unbox(.$key[1]),
label = unbox(.$label[1]),
pos = select(., x, y))
write(toJSON(list(iterations = iterations, data = resultsByObs)), "mnist.json")
#### save plot as animated gif ####
computeLimit <- function(f, cumf) {
cumf(lapply(results, f))
}
xmax <- computeLimit(. %>% {max(abs(.$V2))}, cummax)
ymax <- computeLimit(. %>% {max(abs(.$V3))}, cummax)
plotResult <- function(i) {
ggplot(results[[i]]) +
aes(V2, V3, color = as.factor(V1), label = V1) +
#geom_point() +
geom_text() +
xlim(-xmax[i], xmax[i]) +
ylim(-ymax[i], ymax[i])
}
traceAnimate <- function(n = length(results), step = 1) {
lapply(seq(1, n, step), function(i) {
print(plotResult(i))
})
}
file.remove("tsne.gif")
saveGIF(traceAnimate(step = 5), interval = 0.05, movie.name = "tsne.gif", loop = 1)
================================================
FILE: project/Common.scala
================================================
import sbt._
import Keys._
import com.typesafe.sbt.GitPlugin.autoImport._
import scala.language.experimental.macros
import scala.reflect.macros.Context
object Common {
val commonSettings = Seq(
organization in ThisBuild := "com.github.saurfang",
javacOptions ++= Seq("-source", "1.7", "-target", "1.7"),
scalacOptions ++= Seq("-target:jvm-1.7", "-deprecation", "-feature"),
//git.useGitDescribe := true,
git.baseVersion := "0.0.1",
parallelExecution in test := false,
updateOptions := updateOptions.value.withCachedResolution(true)
)
def tsneProject(path: String): Project = macro tsneProjectMacroImpl
def tsneProjectMacroImpl(c: Context)(path: c.Expr[String]) = {
import c.universe._
reify {
(Project.projectMacroImpl(c).splice in file(path.splice)).
settings(name := path.splice).
settings(Dependencies.Versions).
settings(commonSettings: _*)
}
}
}
================================================
FILE: project/Dependencies.scala
================================================
import sbt._
import Keys._
object Dependencies {
val Versions = Seq(
crossScalaVersions := Seq("2.11.8", "2.10.5"),
scalaVersion := crossScalaVersions.value.head
)
object Compile {
val spark = "org.apache.spark" %% "spark-mllib" % "2.1.0" % "provided"
val breeze_natives = "org.scalanlp" %% "breeze-natives" % "0.11.2" % "provided"
val logging = Seq(
"org.slf4j" % "slf4j-api" % "1.7.16",
"org.slf4j" % "slf4j-log4j12" % "1.7.16")
object Test {
val scalatest = "org.scalatest" %% "scalatest" % "3.0.0" % "test"
}
}
import Compile._
val l = libraryDependencies
val core = l ++= Seq(spark, breeze_natives, Test.scalatest) ++ logging
}
================================================
FILE: project/SparkSubmit.scala
================================================
import sbtsparksubmit.SparkSubmitPlugin.autoImport._
object SparkSubmit {
lazy val settings =
SparkSubmitSetting("sparkMNIST",
Seq(
"--master", "local[3]",
"--class", "com.github.saurfang.spark.tsne.examples.MNIST"
)
)
}
================================================
FILE: project/build.properties
================================================
sbt.version=0.13.13
================================================
FILE: project/plugins.sbt
================================================
addSbtPlugin("com.github.gseitz" % "sbt-release" % "1.0.0")
addSbtPlugin("me.lessis" % "bintray-sbt" % "0.2.1")
addSbtPlugin("com.typesafe.sbt" % "sbt-git" % "0.8.4")
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.13.0")
addSbtPlugin("com.github.saurfang" % "sbt-spark-submit" % "0.0.4")
================================================
FILE: spark-tsne-core/src/main/scala/com/github/saurfang/spark/tsne/TSNEGradient.scala
================================================
package com.github.saurfang.spark.tsne
import breeze.linalg._
import breeze.numerics._
import com.github.saurfang.spark.tsne.tree.SPTree
import org.slf4j.LoggerFactory
object TSNEGradient {
def logger = LoggerFactory.getLogger(TSNEGradient.getClass)
/**
* Compute the numerator from the matrix Y
*
* @param idx the index in the matrix to use.
* @param Y the matrix to analyze
* @return the numerator
*/
def computeNumerator(Y: DenseMatrix[Double], idx: Int *): DenseMatrix[Double] = {
// Y_sum = ||Y_i||^2
val sumY = sum(pow(Y, 2).apply(*, ::)) // n * 1
val subY = Y(idx, ::).toDenseMatrix // k * 1
val y1: DenseMatrix[Double] = Y * (-2.0 :* subY.t) // n * k
val num: DenseMatrix[Double] = (y1(::, *) + sumY).t // k * n
num := 1.0 :/ (1.0 :+ (num(::, *) + sumY(idx).toDenseVector)) // k * n
idx.indices.foreach(i => num.update(i, idx(i), 0.0)) // num(i, i) = 0
num
}
/**
* Compute the TSNE Gradient at i. Update the gradient through dY then return costs attributed at i.
*
* @param data data point for row i by list of pair of (j, p_ij) and 0 <= j < n
* @param Y current Y [n * 2]
* @param totalNum the common numerator that captures the t-distribution of Y
* @param dY gradient of Y
* @return loss attributed to row i
*/
def compute(
data: Array[(Int, Iterable[(Int, Double)])],
Y: DenseMatrix[Double],
num: DenseMatrix[Double],
totalNum: Double,
dY: DenseMatrix[Double],
exaggeration: Boolean): Double = {
// q = (1 + ||Y_i - Y_j||^2)^-1 / sum(1 + ||Y_k - Y_l||^2)^-1
val q: DenseMatrix[Double] = num / totalNum
q.foreachPair{case ((i, j), v) => q.update(i, j, math.max(v, 1e-12))}
// q = q - p
val loss = data.zipWithIndex.flatMap {
case ((_, itr), i) =>
itr.map{
case (j, p) =>
val exaggeratedP = if(exaggeration) p * 4 else p
val qij = q(i, j)
val l = exaggeratedP * math.log(exaggeratedP / qij)
q.update(i, j, qij - exaggeratedP)
if(l.isNaN) 0.0 else l
}
}.sum
// l = [ (p_ij - q_ij) * (1 + ||Y_i - Y_j||^2)^-1 ]
q :*= -num
// l_sum = [0 0 ... sum(l) ... 0]
sum(q(*, ::)).foreachPair{ case (i, v) => q.update(i, data(i)._1, q(i, data(i)._1) - v) }
// dY_i = -4 * (l - l_sum) * Y
val dYi: DenseMatrix[Double] = -4.0 :* (q * Y)
data.map(_._1).zipWithIndex.foreach{
case (i, idx) => dY(i, ::) := dYi(idx, ::)
}
loss
}
/** BH Tree related functions **/
/**
*
* @param data array of (row_id, Seq(col_id), Vector(P_ij))
* @param Y matrix
* @param posF positive forces
*/
def computeEdgeForces(data: Array[(Int, Seq[Int], DenseVector[Double])],
Y: DenseMatrix[Double],
posF: DenseMatrix[Double]): Unit = {
data.foreach {
case (i, cols, vec) =>
// k x D - 1 x D => k x D
val diff = Y(cols, ::).toDenseMatrix.apply(*, ::) - Y(i, ::).t
// k x D => k x 1
val qZ = 1.0 :+ sum(pow(diff, 2).apply(*, ::))
posF(i, ::) := (vec :/ qZ).t * (-diff)
}
}
def computeNonEdgeForces(tree: SPTree,
Y: DenseMatrix[Double],
theta: Double,
negF: DenseMatrix[Double],
idx: Int *): Double = {
idx.foldLeft(0.0)((acc, i) => acc + computeNonEdgeForce(tree, Y(i, ::).t, theta, negF, i))
}
/**
* Calcualte negative forces using BH approximation
*
* @param tree SPTree used for approximation
* @param y y_i
* @param theta threshold for correctness / speed
* @param negF negative forces
* @param i row
* @return sum of Q
*/
private def computeNonEdgeForce(tree: SPTree,
y: DenseVector[Double],
theta: Double,
negF: DenseMatrix[Double],
i: Int): Double = {
import tree._
if(getCount == 0 || (isLeaf && center.equals(y))) {
0.0
} else {
val diff = y - center
val diffSq = sum(pow(diff, 2))
if(isLeaf || radiusSq / diffSq < theta) {
val qZ = 1 / (1 + diffSq)
val nqZ = getCount * qZ
negF(i, ::) :+= (nqZ * qZ * diff).t
nqZ
} else {
children.foldLeft(0.0)((acc, child) => acc + computeNonEdgeForce(child, y, theta, negF, i))
}
}
}
def computeLoss(data: Array[(Int, Seq[Int], DenseVector[Double])],
Y: DenseMatrix[Double],
sumQ: Double): Double = {
data.foldLeft(0.0){
case (acc, (i, cols, vec)) =>
val diff = Y(cols, ::).toDenseMatrix.apply(*, ::) - Y(i, ::).t
val diffSq = sum(pow(diff, 2).apply(*, ::))
val Q = (1.0 :/ (1.0 :+ diffSq)) :/ sumQ
sum(vec :* breeze.numerics.log(max(vec, 1e-12) :/ max(Q, 1e-12)))
}
}
}
================================================
FILE: spark-tsne-core/src/main/scala/com/github/saurfang/spark/tsne/TSNEHelper.scala
================================================
package com.github.saurfang.spark.tsne
import breeze.linalg._
import breeze.stats._
import org.apache.spark.mllib.linalg.distributed.CoordinateMatrix
import org.apache.spark.rdd.RDD
object TSNEHelper {
// p_ij = (p_{i|j} + p_{j|i}) / 2n
def computeP(p_ji: CoordinateMatrix, n: Int): RDD[(Int, Iterable[(Int, Double)])] = {
p_ji.entries
.flatMap(e => Seq(
((e.i.toInt, e.j.toInt), e.value),
((e.j.toInt, e.i.toInt), e.value)
))
.reduceByKey(_ + _) // p + p'
.map{case ((i, j), v) => (i, (j, math.max(v / 2 / n, 1e-12))) } // p / 2n
.groupByKey()
}
/**
* Update Y via gradient dY
* @param Y current Y
* @param dY gradient dY
* @param iY stored y_i - y_{i-1}
* @param gains adaptive learning rates
* @param iteration n
* @param param [[TSNEParam]]
* @return
*/
def update(Y: DenseMatrix[Double],
dY: DenseMatrix[Double],
iY: DenseMatrix[Double],
gains: DenseMatrix[Double],
iteration: Int,
param: TSNEParam): DenseMatrix[Double] = {
import param._
val momentum = if (iteration <= t_momentum) initial_momentum else final_momentum
gains.foreachPair {
case ((i, j), old_gain) =>
val new_gain = math.max(min_gain,
if ((dY(i, j) > 0.0) != (iY(i, j) > 0.0))
old_gain + 0.2
else
old_gain * 0.8
)
gains.update(i, j, new_gain)
val new_iY = momentum * iY(i, j) - eta * new_gain * dY(i, j)
iY.update(i, j, new_iY)
Y.update(i, j, Y(i, j) + new_iY) // Y += iY
}
val t_Y: DenseVector[Double] = mean(Y(::, *)).t
val y_sub = Y(*, ::)
Y := y_sub - t_Y
}
}
================================================
FILE: spark-tsne-core/src/main/scala/com/github/saurfang/spark/tsne/TSNEParam.scala
================================================
package com.github.saurfang.spark.tsne
case class TSNEParam(
early_exaggeration: Int = 100,
exaggeration_factor: Double = 4.0,
t_momentum: Int = 25,
initial_momentum: Double = 0.5,
final_momentum: Double = 0.8,
eta: Double = 500.0,
min_gain: Double = 0.01
)
================================================
FILE: spark-tsne-core/src/main/scala/com/github/saurfang/spark/tsne/X2P.scala
================================================
package com.github.saurfang.spark.tsne
import breeze.linalg.DenseVector
import org.apache.spark.mllib.X2PHelper._
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.linalg.distributed.{CoordinateMatrix, MatrixEntry, RowMatrix}
import org.apache.spark.mllib.rdd.MLPairRDDFunctions._
import org.slf4j.LoggerFactory
object X2P {
private def logger = LoggerFactory.getLogger(X2P.getClass)
def apply(x: RowMatrix, tol: Double = 1e-5, perplexity: Double = 30.0): CoordinateMatrix = {
require(tol >= 0, "Tolerance must be non-negative")
require(perplexity > 0, "Perplexity must be positive")
val mu = (3 * perplexity).toInt //TODO: Expose this as parameter
val logU = Math.log(perplexity)
val norms = x.rows.map(Vectors.norm(_, 2.0))
norms.persist()
val rowsWithNorm = x.rows.zip(norms).map{ case (v, norm) => VectorWithNorm(v, norm) }
val neighbors = rowsWithNorm.zipWithIndex()
.cartesian(rowsWithNorm.zipWithIndex())
.flatMap {
case ((u, i), (v, j)) =>
if(i < j) {
val dist = fastSquaredDistance(u, v)
Seq((i, (j, dist)), (j, (i, dist)))
} else Seq.empty
}
.topByKey(mu)(Ordering.by(e => -e._2))
val p_betas =
neighbors.map {
case (i, arr) =>
var betamin = Double.NegativeInfinity
var betamax = Double.PositiveInfinity
var beta = 1.0
val d = DenseVector(arr.map(_._2))
var (h, p) = Hbeta(d, beta)
//logInfo("data was " + d.toArray.toList)
//logInfo("array P was " + p.toList)
// Evaluate whether the perplexity is within tolerance
def Hdiff = h - logU
var tries = 0
while (Math.abs(Hdiff) > tol && tries < 50) {
//If not, increase or decrease precision
if (Hdiff > 0) {
betamin = beta
beta = if (betamax.isInfinite) beta * 2 else (beta + betamax) / 2
} else {
betamax = beta
beta = if (betamin.isInfinite) beta / 2 else (beta + betamin) / 2
}
// Recompute the values
val HP = Hbeta(d, beta)
h = HP._1
p = HP._2
tries = tries + 1
}
//logInfo("array P is " + p.toList)
(arr.map(_._1).zip(p.toArray).map { case (j, v) => MatrixEntry(i, j, v) }, beta)
}
logger.info("Mean value of sigma: " + p_betas.map(x => math.sqrt(1 / x._2)).mean)
new CoordinateMatrix(p_betas.flatMap(_._1))
}
}
================================================
FILE: spark-tsne-core/src/main/scala/com/github/saurfang/spark/tsne/impl/BHTSNE.scala
================================================
package com.github.saurfang.spark.tsne.impl
import breeze.linalg._
import breeze.stats.distributions.Rand
import com.github.saurfang.spark.tsne.tree.SPTree
import com.github.saurfang.spark.tsne.{TSNEGradient, TSNEHelper, TSNEParam, X2P}
import org.apache.spark.mllib.linalg.distributed.RowMatrix
import org.apache.spark.storage.StorageLevel
import org.slf4j.LoggerFactory
import scala.util.Random
object BHTSNE {
private def logger = LoggerFactory.getLogger(BHTSNE.getClass)
def tsne(
input: RowMatrix,
noDims: Int = 2,
maxIterations: Int = 1000,
perplexity: Double = 30,
theta: Double = 0.5,
reportLoss: Int => Boolean = {i => i % 10 == 0},
callback: (Int, DenseMatrix[Double], Option[Double]) => Unit = {case _ => },
seed: Long = Random.nextLong()
): DenseMatrix[Double] = {
if(input.rows.getStorageLevel == StorageLevel.NONE) {
logger.warn("Input is not persisted and performance could be bad")
}
Rand.generator.setSeed(seed)
val tsneParam = TSNEParam()
import tsneParam._
val n = input.numRows().toInt
val Y: DenseMatrix[Double] = DenseMatrix.rand(n, noDims, Rand.gaussian(0, 1)) :/ 1e4
val iY = DenseMatrix.zeros[Double](n, noDims)
val gains = DenseMatrix.ones[Double](n, noDims)
// approximate p_{j|i}
val p_ji = X2P(input, 1e-5, perplexity)
val P = TSNEHelper.computeP(p_ji, n).glom()
.map(rows => rows.map {
case (i, data) =>
(i, data.map(_._1).toSeq, DenseVector(data.map(_._2 * exaggeration_factor).toArray))
})
.cache()
var iteration = 1
while(iteration <= maxIterations) {
val bcY = P.context.broadcast(Y)
val bcTree = P.context.broadcast(SPTree(Y))
val initialValue = (DenseMatrix.zeros[Double](n, noDims), DenseMatrix.zeros[Double](n, noDims), 0.0)
val (posF, negF, sumQ) = P.treeAggregate(initialValue)(
seqOp = (c, v) => {
// c: (pos, neg, sumQ), v: Array[(i, Seq(j), vec(Distance))]
TSNEGradient.computeEdgeForces(v, bcY.value, c._1)
val q = TSNEGradient.computeNonEdgeForces(bcTree.value, bcY.value, theta, c._2, v.map(_._1): _*)
(c._1, c._2, c._3 + q)
},
combOp = (c1, c2) => {
// c: (grad, loss)
(c1._1 + c2._1, c1._2 + c2._2, c1._3 + c2._3)
})
val dY: DenseMatrix[Double] = posF :- (negF :/ sumQ)
TSNEHelper.update(Y, dY, iY, gains, iteration, tsneParam)
if(reportLoss(iteration)) {
val loss = P.treeAggregate(0.0)(
seqOp = (c, v) => {
TSNEGradient.computeLoss(v, bcY.value, sumQ)
},
combOp = _ + _
)
logger.debug(s"Iteration $iteration finished with $loss")
callback(iteration, Y.copy, Some(loss))
} else {
logger.debug(s"Iteration $iteration finished")
callback(iteration, Y.copy, None)
}
bcY.destroy()
bcTree.destroy()
//undo early exaggeration
if(iteration == early_exaggeration) {
P.foreach {
rows => rows.foreach {
case (_, _, vec) => vec.foreachPair { case (i, v) => vec.update(i, v / exaggeration_factor) }
}
}
}
iteration += 1
}
Y
}
}
================================================
FILE: spark-tsne-core/src/main/scala/com/github/saurfang/spark/tsne/impl/LBFGSTSNE.scala
================================================
package com.github.saurfang.spark.tsne.impl
import breeze.linalg._
import breeze.optimize.{CachedDiffFunction, DiffFunction, LBFGS}
import breeze.stats.distributions.Rand
import com.github.saurfang.spark.tsne.{TSNEGradient, X2P}
import org.apache.spark.mllib.linalg.distributed.RowMatrix
import org.apache.spark.rdd.RDD
import org.apache.spark.storage.StorageLevel
import org.slf4j.LoggerFactory
import scala.util.Random
/**
* TODO: This doesn't work at all (yet or ever).
*/
object LBFGSTSNE {
private def logger = LoggerFactory.getLogger(LBFGSTSNE.getClass)
def tsne(
input: RowMatrix,
noDims: Int = 2,
maxNumIterations: Int = 1000,
numCorrections: Int = 10,
convergenceTol: Double = 1e-4,
perplexity: Double = 30,
seed: Long = Random.nextLong()): DenseMatrix[Double] = {
if(input.rows.getStorageLevel == StorageLevel.NONE) {
logger.warn("Input is not persisted and performance could be bad")
}
Rand.generator.setSeed(seed)
val n = input.numRows().toInt
val early_exaggeration = 100
val t_momentum = 250
val initial_momentum = 0.5
val final_momentum = 0.8
val eta = 500.0
val min_gain = 0.01
val Y: DenseMatrix[Double] = DenseMatrix.rand(n, noDims, Rand.gaussian) //:* .0001
val iY = DenseMatrix.zeros[Double](n, noDims)
val gains = DenseMatrix.ones[Double](n, noDims)
// approximate p_{j|i}
val p_ji = X2P(input, 1e-5, perplexity)
//logInfo(p_ji.toRowMatrix().rows.collect().toList.toString)
// p_ij = (p_{i|j} + p_{j|i}) / 2n
val P = p_ji.transpose().entries.union(p_ji.entries)
.map(e => ((e.i.toInt, e.j.toInt), e.value))
.reduceByKey(_ + _)
.map{case ((i, j), v) => (i, (j, v / 2 / n)) }
.groupByKey()
.glom()
.cache()
var iteration = 1
{
val costFun = new CostFun(P, n, noDims, true)
val lbfgs = new LBFGS[DenseVector[Double]](maxNumIterations, numCorrections, convergenceTol)
val states = lbfgs.iterations(new CachedDiffFunction(costFun), new DenseVector(Y.data))
while (states.hasNext) {
val state = states.next()
val loss = state.value
//logInfo(state.convergedReason.get.toString)
logger.debug(s"Iteration $iteration finished with $loss")
Y := asDenseMatrix(state.x, n, noDims)
//subscriber.onNext((iteration, Y.copy, Some(loss)))
iteration += 1
}
}
{
val costFun = new CostFun(P, n, noDims, false)
val lbfgs = new LBFGS[DenseVector[Double]](maxNumIterations, numCorrections, convergenceTol)
val states = lbfgs.iterations(new CachedDiffFunction(costFun), new DenseVector(Y.data))
while (states.hasNext) {
val state = states.next()
val loss = state.value
//logInfo(state.convergedReason.get.toString)
logger.debug(s"Iteration $iteration finished with $loss")
Y := asDenseMatrix(state.x, n, noDims)
//subscriber.onNext((iteration, Y.copy, Some(loss)))
iteration += 1
}
}
Y
}
private[this] def asDenseMatrix(v: DenseVector[Double], n: Int, noDims: Int) = {
v.asDenseMatrix.reshape(n, noDims)
}
private class CostFun(
P: RDD[Array[(Int, Iterable[(Int, Double)])]],
n: Int,
noDims: Int,
exaggeration: Boolean) extends DiffFunction[DenseVector[Double]] {
override def calculate(weights: DenseVector[Double]): (Double, DenseVector[Double]) = {
val bcY = P.context.broadcast(asDenseMatrix(weights, n, noDims))
val bcExaggeration = P.context.broadcast(exaggeration)
val numerator = P.map{ arr => TSNEGradient.computeNumerator(bcY.value, arr.map(_._1): _*) }.cache()
val bcNumerator = P.context.broadcast({
numerator.treeAggregate(0.0)(seqOp = (x, v) => x + sum(v), combOp = _ + _)
})
val (dY, loss) = P.zip(numerator).treeAggregate((DenseMatrix.zeros[Double](n, noDims), 0.0))(
seqOp = (c, v) => {
// c: (grad, loss), v: (Array[(i, Iterable(j, Distance))], numerator)
// TODO: See if we can include early_exaggeration
val l = TSNEGradient.compute(v._1, bcY.value, v._2, bcNumerator.value, c._1, bcExaggeration.value)
(c._1, c._2 + l)
},
combOp = (c1, c2) => {
// c: (grad, loss)
(c1._1 += c2._1, c1._2 + c2._2)
})
numerator.unpersist()
(loss, new DenseVector(dY.data))
}
}
}
================================================
FILE: spark-tsne-core/src/main/scala/com/github/saurfang/spark/tsne/impl/SimpleTSNE.scala
================================================
package com.github.saurfang.spark.tsne.impl
import breeze.linalg._
import breeze.stats.distributions.Rand
import com.github.saurfang.spark.tsne.{TSNEGradient, TSNEHelper, TSNEParam, X2P}
import org.apache.spark.mllib.linalg.distributed.RowMatrix
import org.apache.spark.storage.StorageLevel
import org.slf4j.LoggerFactory
import scala.util.Random
object SimpleTSNE {
private def logger = LoggerFactory.getLogger(SimpleTSNE.getClass)
def tsne(
input: RowMatrix,
noDims: Int = 2,
maxIterations: Int = 1000,
perplexity: Double = 30,
callback: (Int, DenseMatrix[Double], Option[Double]) => Unit = {case _ => },
seed: Long = Random.nextLong()): DenseMatrix[Double] = {
if(input.rows.getStorageLevel == StorageLevel.NONE) {
logger.warn("Input is not persisted and performance could be bad")
}
Rand.generator.setSeed(seed)
val tsneParam = TSNEParam()
import tsneParam._
val n = input.numRows().toInt
val Y: DenseMatrix[Double] = DenseMatrix.rand(n, noDims, Rand.gaussian(0, 1))
val iY = DenseMatrix.zeros[Double](n, noDims)
val gains = DenseMatrix.ones[Double](n, noDims)
// approximate p_{j|i}
val p_ji = X2P(input, 1e-5, perplexity)
val P = TSNEHelper.computeP(p_ji, n).glom().cache()
var iteration = 1
while(iteration <= maxIterations) {
val bcY = P.context.broadcast(Y)
val numerator = P.map{ arr => TSNEGradient.computeNumerator(bcY.value, arr.map(_._1): _*) }.cache()
val bcNumerator = P.context.broadcast({
numerator.treeAggregate(0.0)(seqOp = (x, v) => x + sum(v), combOp = _ + _)
})
val (dY, loss) = P.zip(numerator).treeAggregate((DenseMatrix.zeros[Double](n, noDims), 0.0))(
seqOp = (c, v) => {
// c: (grad, loss), v: (Array[(i, Iterable(j, Distance))], numerator)
val l = TSNEGradient.compute(v._1, bcY.value, v._2, bcNumerator.value, c._1, iteration <= early_exaggeration)
(c._1, c._2 + l)
},
combOp = (c1, c2) => {
// c: (grad, loss)
(c1._1 + c2._1, c1._2 + c2._2)
})
bcY.destroy()
bcNumerator.destroy()
numerator.unpersist()
TSNEHelper.update(Y, dY, iY, gains, iteration, tsneParam)
logger.debug(s"Iteration $iteration finished with $loss")
callback(iteration, Y.copy, Some(loss))
iteration += 1
}
Y
}
}
================================================
FILE: spark-tsne-core/src/main/scala/com/github/saurfang/spark/tsne/tree/SPTree.scala
================================================
package com.github.saurfang.spark.tsne.tree
import breeze.linalg._
import breeze.numerics._
import scala.annotation.tailrec
class SPTree private[tree](val dimension: Int,
val corner: DenseVector[Double],
val width: DenseVector[Double]) extends Serializable {
private[this] val childWidth: DenseVector[Double] = width :/ 2.0
lazy val radiusSq: Double = sum(pow(width, 2))
private[tree] val totalMass: DenseVector[Double] = DenseVector.zeros(dimension)
private var count: Int = 0
private var leaf: Boolean = true
val center: DenseVector[Double] = DenseVector.zeros(dimension)
lazy val children: Array[SPTree] = {
(0 until pow(2, dimension)).toArray.map {
i =>
val bits = DenseVector(s"%0${dimension}d".format(i.toBinaryString.toInt).toArray.map(_.toDouble - '0'.toDouble))
val childCorner: DenseVector[Double] = corner + (bits :* childWidth)
new SPTree(dimension, childCorner, childWidth)
}
}
final def insert(vector: DenseVector[Double], finalize: Boolean = false): SPTree = {
totalMass += vector
count += 1
if(leaf) {
if(count == 1) { // first to leaf
center := vector
} else if(!vector.equals(center)) {
(1 until count).foreach(_ => getCell(center).insert(center, finalize)) //subdivide
leaf = false
}
}
if(finalize) computeCenter(false)
if(leaf) this else getCell(vector).insert(vector, finalize)
}
def computeCenter(recursive: Boolean = true): Unit = {
if(count > 0) {
center := totalMass / count.toDouble
if(recursive) children.foreach(_.computeCenter())
}
}
def getCell(vector: DenseVector[Double]): SPTree = {
val idx = ((vector - corner) :/ childWidth).data
children(idx.foldLeft(0)((acc, i) => acc * 2 + min(max(i.ceil.toInt - 1, 0), 1)))
}
def getCount: Int = count
def isLeaf: Boolean = leaf
}
object SPTree {
def apply(Y: DenseMatrix[Double]): SPTree = {
val d = Y.cols
val minMaxs = minMax(Y(::, *)).t
val mins = minMaxs.mapValues(_._1)
val maxs = minMaxs.mapValues(_._2)
val tree = new SPTree(Y.cols, mins, maxs - mins)
// insert points but wait till end to compute all centers
//Y(*, ::).foreach(tree.insert(_, finalize = false))
(0 until Y.rows).foreach(i => tree.insert(Y(i, ::).t, finalize = false))
// compute all center of mass
tree.computeCenter()
tree
}
}
================================================
FILE: spark-tsne-core/src/main/scala/org/apache/spark/mllib/X2PHelper.scala
================================================
package org.apache.spark.mllib
import breeze.linalg._
import breeze.numerics._
import org.apache.spark.mllib.linalg.{Vector, Vectors}
import org.apache.spark.mllib.util.MLUtils
object X2PHelper {
case class VectorWithNorm(vector: Vector, norm: Double)
def fastSquaredDistance(v1: VectorWithNorm, v2: VectorWithNorm): Double = {
MLUtils.fastSquaredDistance(v1.vector, v1.norm, v2.vector, v2.norm)
}
def Hbeta(D: DenseVector[Double], beta: Double = 1.0) : (Double, DenseVector[Double]) = {
val P: DenseVector[Double] = exp(- D * beta)
val sumP = sum(P)
if(sumP == 0) {
(0.0, DenseVector.zeros(D.size))
}else {
val H = log(sumP) + (beta * sum(D :* P) / sumP)
(H, P / sumP)
}
}
}
================================================
FILE: spark-tsne-core/src/test/scala/com/github/saurfang/spark/tsne/BugDemonstrationTest.scala
================================================
package com.github.saurfang.spark.tsne
import org.apache.spark.mllib.linalg.{Vectors, Vector}
import org.apache.spark.mllib.stat.{MultivariateStatisticalSummary, Statistics}
import org.apache.spark.sql.SparkSession
import org.scalatest.{BeforeAndAfterAll, FunSuite, Matchers}
/**
* This test demonstrates the bug introduced when upgrading the codebase to spark 2.1.
*
* For completeness and to check regressions, it's now added to the codebase.
*
* @author erwin.vaneijk@gmail.com
*/
class BugDemonstrationTest extends FunSuite with Matchers with BeforeAndAfterAll {
private var sparkSession : SparkSession = _
override def beforeAll(): Unit = {
super.beforeAll()
sparkSession = SparkSession.builder().appName("BugTests").master("local[2]").getOrCreate()
}
override def afterAll(): Unit = {
super.afterAll()
sparkSession.stop()
}
test("This demonstrates a bug was fixed in tsne-spark 2.1") {
val sc = sparkSession.sparkContext
val observations = sc.parallelize(
Seq(
Vectors.dense(1.0, 10.0, 100.0),
Vectors.dense(2.0, 20.0, 200.0),
Vectors.dense(3.0, 30.0, 300.0)
)
)
// Compute column summary statistics.
val summary: MultivariateStatisticalSummary = Statistics.colStats(observations)
val expectedMean = Vectors.dense(2.0,20.0,200.0)
val resultMean = summary.mean
assertEqualEnough(resultMean, expectedMean)
val expectedVariance = Vectors.dense(1.0,100.0,10000.0)
assertEqualEnough(summary.variance, expectedVariance)
val expectedNumNonZeros = Vectors.dense(3.0, 3.0, 3.0)
assertEqualEnough(summary.numNonzeros, expectedNumNonZeros)
}
private def assertEqualEnough(sample: Vector, expected: Vector): Unit = {
expected.toArray.zipWithIndex.foreach{ case(d: Double, i: Int) =>
sample(i) should be (d +- 1E-12)
}
}
}
================================================
FILE: spark-tsne-core/src/test/scala/com/github/saurfang/spark/tsne/TSNEGradientTest.scala
================================================
package com.github.saurfang.spark.tsne
import breeze.linalg._
import org.scalatest.{FunSuite, Matchers}
/**
* Created by forest on 7/17/15.
*/
class TSNEGradientTest extends FunSuite with Matchers {
test("computeNumerator should compute numerator for sub indices") {
val Y = DenseMatrix.create(3, 2, (1 to 6).map(_.toDouble).toArray)
println(Y)
val num = TSNEGradient.computeNumerator(Y, 0, 2)
println(num)
}
}
================================================
FILE: spark-tsne-core/src/test/scala/com/github/saurfang/spark/tsne/X2PSuite.scala
================================================
package com.github.saurfang.spark.tsne
import org.apache.spark.SharedSparkContext
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.linalg.distributed.RowMatrix
import org.scalatest.{FunSuite, Matchers}
/**
* Created by forest on 8/16/15.
*/
class X2PSuite extends FunSuite with SharedSparkContext with Matchers {
test("Test X2P against tsne.jl implementation") {
val input = new RowMatrix(
sc.parallelize(Seq(1 to 3, 4 to 6, 7 to 9, 10 to 12))
.map(x => Vectors.dense(x.map(_.toDouble).toArray))
)
val output = X2P(input, 1e-5, 2).toRowMatrix().rows.collect().map(_.toArray.toList)
println(output.toList)
//output shouldBe List(List(0, .5, .5), List(.5, 0, .5), List(.5, .5, .0))
}
}
================================================
FILE: spark-tsne-core/src/test/scala/com/github/saurfang/spark/tsne/tree/SPTreeSpec.scala
================================================
package com.github.saurfang.spark.tsne.tree
import breeze.linalg._
import org.scalatest.{FunSpec, Matchers}
class SPTreeSpec extends FunSpec with Matchers {
describe("SPTree") {
describe("with 2 dimensions (quadtree)") {
val tree = new SPTree(2, DenseVector(0.0, 0.0), DenseVector(2.0, 4.0))
import tree._
it("should have 4 children") {
children.length shouldBe 4
}
it("each child should have correct width") {
val width = DenseVector(1.0, 2.0)
children.foreach(x => x.width shouldBe width)
}
it("children should have correct corner") {
children.map(_.corner) shouldBe Array(
DenseVector(0.0, 0.0),
DenseVector(0.0, 2.0),
DenseVector(1.0, 0.0),
DenseVector(1.0, 2.0)
)
}
it("getCell should return correct cell") {
getCell(DenseVector(1.0, 1.0)).corner shouldBe DenseVector(0.0, 0.0)
getCell(DenseVector(1.5, 1.5)).corner shouldBe DenseVector(1.0, 0.0)
getCell(DenseVector(2.0, 2.0)).corner shouldBe DenseVector(1.0, 0.0)
getCell(DenseVector(2.0, 2.5)).corner shouldBe DenseVector(1.0, 2.0)
}
it("should be able to be constructed from DenseMatrix") {
val data = Array(
1.0, 1.0, 1.0, 2.0, 1.1, 1.11, 1.11, 1,
3.0, 1.0, 2.0, 2.0, 1.1, 1.11, 1.11, 1
)
val matrix = DenseMatrix.create[Double](data.length / 2, 2, data)
val tree = SPTree(matrix)
tree.getCount shouldBe matrix.rows
tree.children.map(_.getCount).sum shouldBe matrix.rows
tree.center shouldBe DenseVector(data.grouped(matrix.rows).map(x => x.sum / x.length).toArray)
verifyCorrectness(tree)
}
}
}
def verifyCorrectness(tree: SPTree): Unit = {
if(tree.getCount <= 1) tree.isLeaf shouldBe true
if(tree.getCount > 0) tree.center shouldBe (tree.totalMass / tree.getCount.toDouble)
if(tree.isLeaf) {
tree.children.foreach(_.isLeaf shouldBe true)
tree.children.foreach(_.getCount shouldBe 0)
} else {
tree.children.map(_.getCount).sum shouldBe tree.getCount
val totalMassTally = tree.children.foldLeft(DenseVector.zeros[Double](tree.dimension))((acc, t) => acc + t.totalMass)
(0 until tree.dimension).foreach(i => totalMassTally(i) shouldBe (tree.totalMass(i) +- 1e-5))
tree.children.foreach(verifyCorrectness)
}
}
}
================================================
FILE: spark-tsne-core/src/test/scala/org/apache/spark/LocalSparkContext.scala
================================================
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.spark
import _root_.io.netty.util.internal.logging.{InternalLoggerFactory, Slf4JLoggerFactory}
import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach, Suite}
/** Manages a local `sc` {@link SparkContext} variable, correctly stopping it after each test. */
trait LocalSparkContext extends BeforeAndAfterEach with BeforeAndAfterAll { self: Suite =>
@transient var sc: SparkContext = _
override def beforeAll() {
InternalLoggerFactory.setDefaultFactory(new Slf4JLoggerFactory())
super.beforeAll()
}
override def afterEach() {
resetSparkContext()
super.afterEach()
}
def resetSparkContext(): Unit = {
LocalSparkContext.stop(sc)
sc = null
}
}
object LocalSparkContext {
def stop(sc: SparkContext) {
if (sc != null) {
sc.stop()
}
// To avoid Akka rebinding to the same port, since it doesn't unbind immediately on shutdown
System.clearProperty("spark.driver.port")
}
/** Runs `f` by passing in `sc` and ensures that `sc` is stopped. */
def withSpark[T](sc: SparkContext)(f: SparkContext => T): T = {
try {
f(sc)
} finally {
stop(sc)
}
}
}
================================================
FILE: spark-tsne-core/src/test/scala/org/apache/spark/SharedSparkContext.scala
================================================
package org.apache.spark
import org.scalatest.{BeforeAndAfterAll, Suite}
/** Shares a local `SparkContext` between all tests in a suite and closes it at the end */
trait SharedSparkContext extends BeforeAndAfterAll { self: Suite =>
@transient private var _sc: SparkContext = _
def sc: SparkContext = _sc
var conf = new SparkConf(false)
override def beforeAll() {
_sc = new SparkContext("local[4]", "test", conf)
super.beforeAll()
}
override def afterAll() {
LocalSparkContext.stop(_sc)
_sc = null
super.afterAll()
}
}
================================================
FILE: spark-tsne-examples/src/main/resources/log4j.properties
================================================
# Set everything to be logged to the console
log4j.rootCategory=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark-project.jetty=WARN
log4j.logger.org.spark-project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.spark=WARN
log4j.logger.org.apache.spark.mllib=INFO
================================================
FILE: spark-tsne-examples/src/main/scala/com/github/saurfang/spark/tsne/examples/MNIST.scala
================================================
package com.github.saurfang.spark.tsne.examples
import java.io.{BufferedWriter, OutputStreamWriter}
import com.github.saurfang.spark.tsne.impl._
import com.github.saurfang.spark.tsne.tree.SPTree
import org.apache.hadoop.fs.{FileSystem, Path}
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.linalg.distributed.RowMatrix
import org.apache.spark.{SparkConf, SparkContext}
import org.slf4j.LoggerFactory
object MNIST {
private def logger = LoggerFactory.getLogger(MNIST.getClass)
def main (args: Array[String]) {
val conf = new SparkConf()
.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.registerKryoClasses(Array(classOf[SPTree]))
val sc = new SparkContext(conf)
val hadoopConf = sc.hadoopConfiguration
val fs = FileSystem.get(hadoopConf)
val dataset = sc.textFile("data/MNIST/mnist.csv.gz")
.zipWithIndex()
.filter(_._2 < 6000)
.sortBy(_._2, true, 60)
.map(_._1)
.map(_.split(","))
.map(x => (x.head.toInt, x.tail.map(_.toDouble)))
.cache()
//logInfo(dataset.collect.map(_._2.toList).toList.toString)
//val features = dataset.map(x => Vectors.dense(x._2))
//val scaler = new StandardScaler(true, true).fit(features)
//val scaledData = scaler.transform(features)
// .map(v => Vectors.dense(v.toArray.map(x => if(x.isNaN || x.isInfinite) 0.0 else x)))
// .cache()
val data = dataset.flatMap(_._2)
val mean = data.mean()
val std = data.stdev()
val scaledData = dataset.map(x => Vectors.dense(x._2.map(v => (v - mean) / std))).cache()
val labels = dataset.map(_._1).collect()
val matrix = new RowMatrix(scaledData)
val pcaMatrix = matrix.multiply(matrix.computePrincipalComponents(50))
pcaMatrix.rows.cache()
val costWriter = new BufferedWriter(new OutputStreamWriter(fs.create(new Path(s".tmp/MNIST/cost.txt"), true)))
//SimpleTSNE.tsne(pcaMatrix, perplexity = 20, maxIterations = 200)
BHTSNE.tsne(pcaMatrix, maxIterations = 500, callback = {
//LBFGSTSNE.tsne(pcaMatrix, perplexity = 10, maxNumIterations = 500, numCorrections = 10, convergenceTol = 1e-8)
case (i, y, loss) =>
if(loss.isDefined) logger.info(s"$i iteration finished with loss $loss")
val os = fs.create(new Path(s".tmp/MNIST/result${"%05d".format(i)}.csv"), true)
val writer = new BufferedWriter(new OutputStreamWriter(os))
try {
(0 until y.rows).foreach {
row =>
writer.write(labels(row).toString)
writer.write(y(row, ::).inner.toArray.mkString(",", ",", "\n"))
}
if(loss.isDefined) costWriter.write(loss.get + "\n")
} finally {
writer.close()
}
})
costWriter.close()
sc.stop()
}
}
================================================
FILE: spark-tsne-player/src/main/html/tsne.html
================================================
<!DOCTYPE html>
<html>
<meta charset='utf-8'>
<title>t-SNE Viewer</title>
<style>
#chart {
margin-left: -40px;
height: 650px;
}
text {
font: 10px sans-serif;
}
.axis path, .axis line {
fill: none;
stroke: #000;
shape-rendering: crispEdges;
}
.label {
fill: #777;
}
.iteration.label {
font: 500 196px 'Helvetica Neue';
fill: #ddd;
}
.iteration.label.active {
fill: #aaa;
}
.overlay {
fill: none;
pointer-events: all;
cursor: ew-resize;
}
</style>
<h1>T-SNE Viewer</h1>
<p id='chart'></p>
<aside>Mouseover the iteration to move forward and backwards through time.</aside>
<p class='attribution'>Source: <a href='http://bost.ocks.org/mike/nations'>The Wealth & Health of Nations</a>, <a href='http://bost.ocks.org/mike/'>Mike Bostock</a>.</p>
<script src='https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.6/d3.min.js'></script>
<script src='tsne-json.js'></script>
<script>
// Various accessors that specify the four dimensions of data to visualize.
function x(d) { return d.x; }
function y(d) { return d.y; }
function text(d) { return d.label; }
function color(d) { return d.label; }
function key(d) { return d.key; }
// Chart dimensions.
var margin = {top: 19.5, right: 19.5, bottom: 19.5, left: 99.5},
width = 640 - margin.right,
height = 640 - margin.top - margin.bottom;
// Various scales. These domains make assumptions of data, naturally.
var xScale = d3.scale.linear().domain([-20, 20]).range([0, width]),
yScale = d3.scale.linear().domain([-20, 20]).range([height, 0]),
colorScale = d3.scale.category10();
// The x & y axes.
var xAxis = d3.svg.axis().orient('bottom').scale(xScale).ticks(12, d3.format(',d')),
yAxis = d3.svg.axis().orient('left').scale(yScale).ticks(12, d3.format(',d'));
// Create the SVG container and set the origin.
var svg = d3.select('#chart').append('svg')
.attr('width', width + margin.left + margin.right)
.attr('height', height + margin.top + margin.bottom)
.append('g')
.attr('transform', 'translate(' + margin.left + ',' + margin.top + ')');
// Add the x-axis.
svg.append('g')
.attr('class', 'x axis')
.attr('transform', 'translate(0,' + height + ')')
.call(xAxis);
// Add the y-axis.
svg.append('g')
.attr('class', 'y axis')
.call(yAxis);
// Add an x-axis label.
svg.append('text')
.attr('class', 'x label')
.attr('text-anchor', 'end')
.attr('x', width)
.attr('y', height - 6);
// Add a y-axis label.
svg.append('text')
.attr('class', 'y label')
.attr('text-anchor', 'end')
.attr('y', 6)
.attr('dy', '.75em')
.attr('transform', 'rotate(-90)');
// Add the iteration label; the value is set on transition.
var label = svg.append('text')
.attr('class', 'iteration label')
.attr('text-anchor', 'end')
.attr('y', height - 24)
.attr('x', width)
.text('0000');
// Load the data.
d3.json('mnist.json?nocache=' + (new Date()).getTime(), function(results) {
var iterations = tsneData.iterations,
maxIteration = iterations.max(),
results = tsneData.data,
animationDuration = 100;
// Add a dot per observation. Initialize the data at 1, and set the colors.
var dot = svg.append('g')
.attr('class', 'dots')
.selectAll('.dot')
.data(interpolateData(1))
.enter().append('text')
.attr('class', 'dot')
.text(function(d) { return text(d); })
.style('fill', function(d) { return colorScale(color(d)); })
.call(position);
// Add an overlay for the iteration label.
var box = label.node().getBBox();
var overlay = svg.append('rect')
.attr('class', 'overlay')
.attr('x', box.x)
.attr('y', box.y)
.attr('width', box.width)
.attr('height', box.height)
.on('mouseover', enableInteraction);
// Start a transition that interpolates the data based on iteration.
svg.transition()
.duration(animationDuration * maxIteration)
.ease('linear')
.tween('iteration', tweenIteration)
.each('end', enableInteraction);
// Positions the dots based on data.
function position(dot) {
dot .attr('x', function(d) { return xScale(x(d)); })
.attr('y', function(d) { return yScale(y(d)); });
}
// After the transition finishes, you can mouseover to change the iteration.
function enableInteraction() {
var iterationScale = d3.scale.linear()
.domain([1, maxIteration])
.range([box.x + 10, box.x + box.width - 10])
.clamp(true);
// Cancel the current transition, if any.
svg.transition().duration(0);
overlay
.on('mouseover', mouseover)
.on('mouseout', mouseout)
.on('mousemove', mousemove)
.on('touchmove', mousemove);
function mouseover() {
label.classed('active', true);
}
function mouseout() {
label.classed('active', false);
}
function mousemove() {
displayIteration(iterationScale.invert(d3.mouse(this)[0]));
}
}
// Tweens the entire chart by first tweening the iteration, and then the data.
// For the interpolated data, the dots and label are redrawn.
function tweenIteration() {
var iteration = d3.interpolateNumber(1, maxIteration);
return function(t) { displayIteration(iteration(t)); };
}
// Updates the display to show the specified iteration.
function displayIteration(iteration) {
var interpolated = interpolateData(iteration, iterations, results);
var limits = interpolated.reduce(function(acc, data) {
return [Math.max(acc[0], Math.abs(data.x)), Math.max(acc[1], Math.abs(data.y))];
}, [0, 0]);
updateScale([-limits[0], limits[0]], [-limits[1], limits[1]]);
dot.data(interpolated, key).call(position);
label.text(pad(Math.round(iteration), 4));
}
});
</script>
gitextract_xem_b0op/
├── .gitattributes
├── .gitignore
├── .travis.yml
├── LICENSE
├── README.md
├── build.sbt
├── data/
│ └── mnist/
│ └── tsne.R
├── project/
│ ├── Common.scala
│ ├── Dependencies.scala
│ ├── SparkSubmit.scala
│ ├── build.properties
│ └── plugins.sbt
├── spark-tsne-core/
│ └── src/
│ ├── main/
│ │ └── scala/
│ │ ├── com/
│ │ │ └── github/
│ │ │ └── saurfang/
│ │ │ └── spark/
│ │ │ └── tsne/
│ │ │ ├── TSNEGradient.scala
│ │ │ ├── TSNEHelper.scala
│ │ │ ├── TSNEParam.scala
│ │ │ ├── X2P.scala
│ │ │ ├── impl/
│ │ │ │ ├── BHTSNE.scala
│ │ │ │ ├── LBFGSTSNE.scala
│ │ │ │ └── SimpleTSNE.scala
│ │ │ └── tree/
│ │ │ └── SPTree.scala
│ │ └── org/
│ │ └── apache/
│ │ └── spark/
│ │ └── mllib/
│ │ └── X2PHelper.scala
│ └── test/
│ └── scala/
│ ├── com/
│ │ └── github/
│ │ └── saurfang/
│ │ └── spark/
│ │ └── tsne/
│ │ ├── BugDemonstrationTest.scala
│ │ ├── TSNEGradientTest.scala
│ │ ├── X2PSuite.scala
│ │ └── tree/
│ │ └── SPTreeSpec.scala
│ └── org/
│ └── apache/
│ └── spark/
│ ├── LocalSparkContext.scala
│ └── SharedSparkContext.scala
├── spark-tsne-examples/
│ └── src/
│ └── main/
│ ├── resources/
│ │ └── log4j.properties
│ └── scala/
│ └── com/
│ └── github/
│ └── saurfang/
│ └── spark/
│ └── tsne/
│ └── examples/
│ └── MNIST.scala
└── spark-tsne-player/
└── src/
└── main/
└── html/
└── tsne.html
Condensed preview — 30 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (63K chars).
[
{
"path": ".gitattributes",
"chars": 127,
"preview": "\n*.gz filter=lfs diff=lfs merge=lfs -text\n*.gif filter=lfs diff=lfs merge=lfs -text\n*.json filter=lfs diff=lfs merge=lfs"
},
{
"path": ".gitignore",
"chars": 72,
"preview": "/RUNNING_PID\n/logs/\nproject/project/\nproject/target/\ntarget/\n.idea\n.tmp\n"
},
{
"path": ".travis.yml",
"chars": 93,
"preview": "language: scala\nscala:\n - 2.10.6\n - 2.11.7\njdk:\n - oraclejdk8\n - oraclejdk7\n - openjdk7\n"
},
{
"path": "LICENSE",
"chars": 11592,
"preview": " Apache License\n Version 2.0, January 2004\n "
},
{
"path": "README.md",
"chars": 1476,
"preview": "# spark-tsne\n\n[](http"
},
{
"path": "build.sbt",
"chars": 461,
"preview": "import Common._\n\nlazy val root = Project(\"spark-tsne\", file(\".\")).\n settings(commonSettings: _*).\n aggregate(core, vis"
},
{
"path": "data/mnist/tsne.R",
"chars": 1632,
"preview": "library(dplyr)\nlibrary(ggplot2)\nlibrary(animation)\nlibrary(jsonlite)\n\nresultFiles <- list.files(\"~/GitHub/spark-tsne/.tm"
},
{
"path": "project/Common.scala",
"chars": 934,
"preview": "import sbt._\nimport Keys._\nimport com.typesafe.sbt.GitPlugin.autoImport._\n\nimport scala.language.experimental.macros\nimp"
},
{
"path": "project/Dependencies.scala",
"chars": 698,
"preview": "import sbt._\nimport Keys._\n\nobject Dependencies {\n val Versions = Seq(\n crossScalaVersions := Seq(\"2.11.8\", \"2.10.5\""
},
{
"path": "project/SparkSubmit.scala",
"chars": 260,
"preview": "import sbtsparksubmit.SparkSubmitPlugin.autoImport._\n\nobject SparkSubmit {\n lazy val settings =\n SparkSubmitSetting("
},
{
"path": "project/build.properties",
"chars": 20,
"preview": "sbt.version=0.13.13\n"
},
{
"path": "project/plugins.sbt",
"chars": 295,
"preview": "addSbtPlugin(\"com.github.gseitz\" % \"sbt-release\" % \"1.0.0\")\n\naddSbtPlugin(\"me.lessis\" % \"bintray-sbt\" % \"0.2.1\")\n\naddSbt"
},
{
"path": "spark-tsne-core/src/main/scala/com/github/saurfang/spark/tsne/TSNEGradient.scala",
"chars": 5012,
"preview": "package com.github.saurfang.spark.tsne\n\nimport breeze.linalg._\nimport breeze.numerics._\nimport com.github.saurfang.spark"
},
{
"path": "spark-tsne-core/src/main/scala/com/github/saurfang/spark/tsne/TSNEHelper.scala",
"chars": 1715,
"preview": "package com.github.saurfang.spark.tsne\n\nimport breeze.linalg._\nimport breeze.stats._\nimport org.apache.spark.mllib.linal"
},
{
"path": "spark-tsne-core/src/main/scala/com/github/saurfang/spark/tsne/TSNEParam.scala",
"chars": 435,
"preview": "package com.github.saurfang.spark.tsne\n\ncase class TSNEParam(\n early_exaggeration: Int = 100,\n "
},
{
"path": "spark-tsne-core/src/main/scala/com/github/saurfang/spark/tsne/X2P.scala",
"chars": 2551,
"preview": "package com.github.saurfang.spark.tsne\n\nimport breeze.linalg.DenseVector\nimport org.apache.spark.mllib.X2PHelper._\nimpor"
},
{
"path": "spark-tsne-core/src/main/scala/com/github/saurfang/spark/tsne/impl/BHTSNE.scala",
"chars": 3401,
"preview": "package com.github.saurfang.spark.tsne.impl\n\nimport breeze.linalg._\nimport breeze.stats.distributions.Rand\nimport com.gi"
},
{
"path": "spark-tsne-core/src/main/scala/com/github/saurfang/spark/tsne/impl/LBFGSTSNE.scala",
"chars": 4631,
"preview": "package com.github.saurfang.spark.tsne.impl\n\nimport breeze.linalg._\nimport breeze.optimize.{CachedDiffFunction, DiffFunc"
},
{
"path": "spark-tsne-core/src/main/scala/com/github/saurfang/spark/tsne/impl/SimpleTSNE.scala",
"chars": 2488,
"preview": "package com.github.saurfang.spark.tsne.impl\n\nimport breeze.linalg._\nimport breeze.stats.distributions.Rand\nimport com.gi"
},
{
"path": "spark-tsne-core/src/main/scala/com/github/saurfang/spark/tsne/tree/SPTree.scala",
"chars": 2433,
"preview": "package com.github.saurfang.spark.tsne.tree\n\nimport breeze.linalg._\nimport breeze.numerics._\n\nimport scala.annotation.ta"
},
{
"path": "spark-tsne-core/src/main/scala/org/apache/spark/mllib/X2PHelper.scala",
"chars": 734,
"preview": "package org.apache.spark.mllib\n\nimport breeze.linalg._\nimport breeze.numerics._\nimport org.apache.spark.mllib.linalg.{Ve"
},
{
"path": "spark-tsne-core/src/test/scala/com/github/saurfang/spark/tsne/BugDemonstrationTest.scala",
"chars": 1867,
"preview": "package com.github.saurfang.spark.tsne\n\nimport org.apache.spark.mllib.linalg.{Vectors, Vector}\nimport org.apache.spark.m"
},
{
"path": "spark-tsne-core/src/test/scala/com/github/saurfang/spark/tsne/TSNEGradientTest.scala",
"chars": 435,
"preview": "package com.github.saurfang.spark.tsne\n\nimport breeze.linalg._\nimport org.scalatest.{FunSuite, Matchers}\n\n/**\n * Created"
},
{
"path": "spark-tsne-core/src/test/scala/com/github/saurfang/spark/tsne/X2PSuite.scala",
"chars": 753,
"preview": "package com.github.saurfang.spark.tsne\n\nimport org.apache.spark.SharedSparkContext\nimport org.apache.spark.mllib.linalg."
},
{
"path": "spark-tsne-core/src/test/scala/com/github/saurfang/spark/tsne/tree/SPTreeSpec.scala",
"chars": 2415,
"preview": "package com.github.saurfang.spark.tsne.tree\n\nimport breeze.linalg._\nimport org.scalatest.{FunSpec, Matchers}\n\nclass SPTr"
},
{
"path": "spark-tsne-core/src/test/scala/org/apache/spark/LocalSparkContext.scala",
"chars": 1962,
"preview": "/*\n * Licensed to the Apache Software Foundation (ASF) under one or more\n * contributor license agreements. See the NOT"
},
{
"path": "spark-tsne-core/src/test/scala/org/apache/spark/SharedSparkContext.scala",
"chars": 559,
"preview": "package org.apache.spark\n\nimport org.scalatest.{BeforeAndAfterAll, Suite}\n\n/** Shares a local `SparkContext` between all"
},
{
"path": "spark-tsne-examples/src/main/resources/log4j.properties",
"chars": 708,
"preview": "# Set everything to be logged to the console\nlog4j.rootCategory=INFO, console\nlog4j.appender.console=org.apache.log4j.Co"
},
{
"path": "spark-tsne-examples/src/main/scala/com/github/saurfang/spark/tsne/examples/MNIST.scala",
"chars": 2807,
"preview": "package com.github.saurfang.spark.tsne.examples\n\n\nimport java.io.{BufferedWriter, OutputStreamWriter}\n\nimport com.github"
},
{
"path": "spark-tsne-player/src/main/html/tsne.html",
"chars": 5784,
"preview": "<!DOCTYPE html>\n<html>\n<meta charset='utf-8'>\n<title>t-SNE Viewer</title>\n<style>\n\n#chart {\n margin-left: -40px;\n heig"
}
]
About this extraction
This page contains the full source code of the saurfang/spark-tsne GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 30 files (57.0 KB), approximately 16.9k tokens. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.