Repository: unibas-gravis/basel-face-pipeline Branch: master Commit: edf583442af9 Files: 22 Total size: 132.6 KB Directory structure: gitextract_eb7rw3gs/ ├── LICENSE.txt ├── README.md ├── build.sbt ├── pipeline-data/ │ └── .gitignore └── src/ └── main/ ├── resources/ │ └── logback.xml └── scala/ ├── ch/ │ └── unibas/ │ └── cs/ │ └── gravis/ │ └── facepipeline/ │ ├── BU3DDataProvider.scala │ ├── DataProvider.scala │ └── PipelineStep.scala ├── fitting/ │ ├── StandardFitScript.scala │ └── experiments/ │ ├── QualitativeLFW.scala │ └── RecognitionMultiPIE.scala ├── modelbuilding/ │ └── ModelBuilding.scala ├── preprocessing/ │ ├── ConvertBu3DRawData.scala │ └── PrepareReferenceLandmarks.scala └── registration/ ├── Registration.scala ├── experiments/ │ └── Bu3DFELandmarkEvaluation.scala ├── metrics/ │ └── HuberDistanceMetric.scala ├── modelbuilding/ │ ├── BuildCoreExpressionPrior.scala │ ├── BuildNeutralPrior.scala │ ├── FaceKernel.scala │ └── FaceMask.scala └── utils/ └── VisualLogger.scala ================================================ FILE CONTENTS ================================================ ================================================ FILE: LICENSE.txt ================================================ Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "{}" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright {yyyy} {name of copyright owner} Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ================================================ FILE: README.md ================================================ # Basel Face Registration Pipeline This repository contains all the code to reproduce our results from our recent publication: - Thomas Gerig, Andreas Morel-Forster, Clemens Blumer, Bernhard Egger, Marcel Lüthi, Sandro Schönborn and Thomas Vetter "Morphable Face Models - An Open Framework" ( [edoc-unibas](https://edoc.unibas.ch/69084/) / [IEEE](https://ieeexplore.ieee.org/document/8373814) ) IN: 13th IEEE Conference on Automatic Face and Gesture Recognition (FG 2018) pp. 75-82 ## Overview After the following information, we list all nescessary steps that you need to take to aquire and prepare the data as well as to run the pipeline and the experiments in the next sections. ### Problems under Windows When you expericence some problems under Windows while importing the data, please use the branch **updateScalismoFaces**. ### Reporting problems and discussion When you experience problems, you have questions or feedback please use the mailing list for [Morphable face models - an open framework](https://groups.google.com/forum/#!categories/scalismo-faces/morphable-face-models---an-open-framework). ## Preparation ### Step 1: Folder structure and Basel reference mesh For the registration pipeline and the experiments to work properly, some data, such as reference templates and landmarks are needed. The files are available for download at [Registration Pipeline Data](https://faces.dmi.unibas.ch/bfm/bfm2017.html). The download contains the following in detail: * Manually clicked landmarks for the BU3D-FE database. * BFM reference mesh and expression means. * Landmarks of the reference mesh. * Region mask for model-building. You can copy the content of the zip folder into `pipeline-data`. The coarse structure looks the following: ``` pipeline-data ├── data │ ├── incoming │ ├── bu3dfe │ │ ├── original ├── recognition-experiment ``` If needed, you can change the location of the `pipeline-data` directory in the BU3DDataProvider.scala file. ### Step 2: Bu3DFE Database To register the BU-3DFE you have to acquire the dataset here: [BU-3DFE](http://www.cs.binghamton.edu/~lijun/Research/3DFE/3DFE_Analysis.html) and copy the `/original` folder to `data/bu3dfe/original/`. ### Step 3: Sbt (Scala build tool) We assume that you have sbt already installed. If not, please follow the instructions given [here](http://www.scala-sbt.org/release/tutorial/Setup.html). Generally you can run the code using SBT. An example is how to run it in the terminal with: ``` cd /code-directory/ sbt run ``` If you do not have enough memory use: ``` sbt -J-Xmx50g run ``` Then the different steps are then listed and can be executed by entering the number of the script or by using: ``` sbt "run-main package.Classname" ``` ## Running the Pipeline ### Step 0: Data pre-processing & Folder Structure Creation During the pipeline we do not use the BU3DFE database data directly but first convert the data to match our formats. This step is done only once as a pre-processing and the output can be reused whenever you run a new registration. To convert the original data from the BU3DFE database to our format use the command: ``` sbt "run-main preprocessing.ConvertBu3DRawData" ``` Explain raw data preprocessing steps in script. (The script might need some cleanup.) ### Step 1: Building the Neutral Prior Model Pre-computing the neutral prior model can take quite some time. However, it has to be computed only once offline and is stored in `pipeline-data/data/incoming/reference/gpmodels/`. You can run the building process with: ``` sbt "run-main registration.BuildNeutralPrior" ``` ### Step 2: Building the Core Expression Model The core expression model augments the neutral model with expression deformations. ``` sbt "run-main registration.BuildCoreExpressionModel" ``` ### Step 3: Preprocess Landmarks This step is used to transform the reference landmarks to the new mean of the generated models and to change the uncertainty of the individual landmarks. ``` sbt "run-main preprocessing.PrepareReferenceLandmarks" ``` ### Step 4: Registration ``` sbt -J-Xmx40g "run-main registration.Registration" ``` ### Step 5: Building the Morphable Model The model building contains two steps: - First for each registration result the color is extracted using the input mesh. - Based on all meshes with color a model containing shape, color and expression variations is built. This process may need some time and memory. Once the first step, the color extraction is computed it can be reused if you change for example the mask of the model that should be built. But to change this you have to out comment the corresponding line in the source code. ``` sbt -mem 40000 "run-main modelbuilding.ModelBuilding" ``` ## Face Reconstruction from 2D Image First you have to download the Multi-PIE database and copy the necessary files to the correct folders. This is described in the README file in the folder recogniton-experiment (comes with seperate download of the Basel Face Pipeline [Data](https://faces.dmi.unibas.ch/bfm/bfm2017.html)). For those experiments you need the Basel Face Model 2009 and 2017, which can be downloaded at: [Probabilistic Morphable Models](https://gravis.dmi.unibas.ch/PMM/) To run the 3D reconstructions from the Multi-PIE database, you may want to execute it multiple times in parallel since a single fit taks ~20 minutes: ``` sbt -mem 5000 "fitting.experiments.RecognitionMultiPiePose" ``` And to calculate the recognition scores execute: ``` sbt -mem 5000 "fitting.experiments.RecognitionEvaluation" ``` Those where the neutral scores. To perform the expression experiments, run: ``` sbt -mem 5000 "fitting.experiments.RecognitionMultiPieExpression" sbt -mem 5000 "fitting.experiments.RecognitionEvaluationEx" ``` ================================================ FILE: build.sbt ================================================ name := "basel-face-pipeline" version := "0.1" scalaVersion := "2.11.8" scalacOptions := Seq("-unchecked", "-deprecation", "-encoding", "utf8") resolvers += Resolver.jcenterRepo resolvers += Resolver.bintrayRepo("unibas-gravis", "maven") libraryDependencies += "ch.unibas.cs.gravis" %% "scalismo-faces" % "0.5.0" libraryDependencies += "ch.unibas.cs.gravis" %% "scalismo-ui" % "0.11.+" libraryDependencies += "com.github.tototoshi" %% "scala-csv" % "1.3.3" libraryDependencies += "com.typesafe.scala-logging" %% "scala-logging" % "3.5.0" libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.1.7" libraryDependencies ~= { _.map(_.exclude("org.slf4j", "slf4j-nop")) } ================================================ FILE: pipeline-data/.gitignore ================================================ ================================================ FILE: src/main/resources/logback.xml ================================================ %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n /tmp/face-registration.log false %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n ================================================ FILE: src/main/scala/ch/unibas/cs/gravis/facepipeline/BU3DDataProvider.scala ================================================ /* * Copyright University of Basel, Graphics and Vision Research Group * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ package ch.unibas.cs.gravis.facepipeline import java.io.File import registration.modelbuilding.FaceMask import scalismo.faces.io.{MoMoIO, TLMSLandmarksIO} import scalismo.faces.landmarks.TLMSLandmark3D import scalismo.faces.mesh.{BinaryMask, ColorNormalMesh3D} import scalismo.faces.momo.MoMo import scalismo.geometry.{Landmark, _3D} import scalismo.io.{LandmarkIO, MeshIO, StatismoIO} import scalismo.mesh.TriangleMesh import scalismo.statisticalmodel.StatisticalMeshModel import scala.io.Source import scala.reflect.io.Path import scala.util.{Failure, Success, Try} object BU3DDataProvider extends DataProvider { override case object Neutral extends ExpressionType { override def toString: String = "_NE00" } case object Sadness extends ExpressionType { override def toString: String = "_SA04" } case object Surprise extends ExpressionType { override def toString: String = "_SU04" } case object Disgust extends ExpressionType { override def toString: String = "_DI04" } case object Fear extends ExpressionType { override def toString: String = "_FE04" } case object Joy extends ExpressionType { override def toString: String = "_HA04" } case object Anger extends ExpressionType { override def toString: String = "_AN04" } case object CoreExpression extends ExpressionType { override def toString: String = "_ALLEXP" } object Expressions { def expressionList(): Seq[ExpressionType] = Seq(Neutral, Sadness, Surprise, Disgust, Fear, Joy, Anger) def expressionModelTypes(): Seq[ExpressionType] = Seq(Neutral,CoreExpression) } override def expressions() = Expressions.expressionList() case object RAW extends MaskType { override def toString: String = "_RAW" } case object F3D extends MaskType { override def toString: String = "_F3D" } object Masks { def maskList(): Seq[MaskType] = Seq(RAW, F3D) } override def masks: Seq[MaskType] = Masks.maskList() case class BU3DID(override val id: String, override val raceTag: String) extends Person object BU3DID { def fromFilename(filename: String): BU3DID = { BU3DID(filename.substring(0, 5), filename.substring(10, 12)) } } override def personFromFilename(filename: String): Person = BU3DID.fromFilename(filename) case object Basel extends DataFlag { override def toString: String = "_basel" } case object Original extends DataFlag { override def toString: String = "" } case object Aligned extends DataFlag { override def toString: String = "_aligned" } object Flags { def lmFlagList(): Seq[DataFlag] = Seq(Basel, Original, Aligned) } private def setFileAccessMode(filename: String): Unit = setFileAccessMode(new File(filename)) private def setFileAccessMode(path: Path): Unit = setFileAccessMode(path.jfile) private def setFileAccessMode(file: File): Unit = { file.setReadable(true,false) file.setWritable(true,false) } override def repositoryRoot: Path = Path("pipeline-data/") override def incoming: BU3DDataProvider.Incoming = { new BU3DDataProvider.Incoming { val incomingPath = repositoryRoot / "data" / "incoming" incomingPath.jfile.mkdirs() override def reference: BU3DDataProvider.Reference = new BU3DDataProvider.Reference { val referencePath = incomingPath / "reference" referencePath.jfile.mkdirs() override def loadFaceMask(): Try[FaceMask] = { val maskPath = referencePath / "masks" maskPath.jfile.mkdirs() for { level_mask <- MeshIO.readScalarMeshField[Int](new File(maskPath.jfile, "level-mask-l7.vtk")) semantic_mask <- MeshIO.readScalarMeshField[Short](new File(maskPath.jfile, "semantic-mask-l7.vtk")).map(_.map(_.toInt)) } yield { FaceMask(level_mask,semantic_mask) } } override def loadMesh(expression: ExpressionType): Try[TriangleMesh[_3D]] = { import scalismo.faces.io.MeshIO val mshPath = referencePath / "mesh" mshPath.jfile.mkdirs() expression match { case Neutral => MeshIO.read(new File(mshPath.jfile, "mean2012_l7_bfm_nomouth.ply")) .map(_.shape) case Sadness => MeshIO.read(new File(mshPath.jfile, "mean2015.1_l7_bfm_nomouth-sadness.ply")) .map(_.shape) case Surprise => MeshIO.read(new File(mshPath.jfile, "mean2015.1_l7_bfm_nomouth-surprise.ply")) .map(_.shape) case Disgust => MeshIO.read(new File(mshPath.jfile, "mean2015.1_l7_bfm_nomouth-disgust.ply")) .map(_.shape) case Fear => MeshIO.read(new File(mshPath.jfile, "mean2015.1_l7_bfm_nomouth-fear.ply")) .map(_.shape) case Joy => MeshIO.read(new File(mshPath.jfile, "mean2015.1_l7_bfm_nomouth-joy.ply")) .map(_.shape) case Anger => MeshIO.read(new File(mshPath.jfile, "mean2015.1_l7_bfm_nomouth-anger.ply")) .map(_.shape) } } override def loadLandmarks(expression: ExpressionType): Try[Seq[Landmark[_3D]]] = { val lmPath = referencePath / "landmarks" lmPath.jfile.mkdirs() LandmarkIO.readLandmarksJson[_3D](new File(lmPath.jfile, s"reference${expression.toString}.json")) } override def saveLandmarks(expression: ExpressionType, landmarks: Seq[Landmark[_3D]]): Try[Unit] = { val lmPath = referencePath / "landmarks" lmPath.jfile.mkdirs() val res = LandmarkIO.writeLandmarksJson[_3D](landmarks.toIndexedSeq, new File(lmPath.jfile, s"reference${expression.toString}.json")) res match { case Success(_) => setFileAccessMode(lmPath) case _ => } res } override def loadLineLandmarks(expression: ExpressionType): Try[Seq[Landmark[_3D]]] = ??? } def landmarksPath(id: Person, expression: ExpressionType, mask: MaskType = RAW, flag: DataFlag = Basel): Path = { incomingPath / "landmarks" / s"${id.id}$expression${id.raceTag}${mask}$flag.tlms" } override def loadLandmarks(id: Person, expression: ExpressionType): Try[Seq[Landmark[_3D]]] = loadLandmarks(id, expression, RAW, Basel) override def loadLandmarks(id: Person, expression: ExpressionType, mask: MaskType): Try[Seq[Landmark[_3D]]] = loadLandmarks(id, expression, mask, Basel) override def loadLandmarks(id: Person, expression: ExpressionType, mask: MaskType, flag: DataFlag): Try[Seq[Landmark[_3D]]] = { val path = landmarksPath(id, expression, mask, flag) TLMSLandmarksIO.read3D(path.jfile) match { case Success(tlmsLandmarks) => Success(tlmsLandmarks.filter(_.visible).map(_.toLandmark)) case Failure(t) => Failure(t) } } override def saveLandmarks(id: Person, expression: ExpressionType, landmarks: Seq[Landmark[_3D]]): Try[Unit] = saveLandmarks(id, expression, RAW, Basel, landmarks) override def saveLandmarks(id: Person, expression: ExpressionType, mask: MaskType, landmarks: Seq[Landmark[_3D]]): Try[Unit] = saveLandmarks(id, expression, mask, Basel, landmarks) override def saveLandmarks(id: Person, expression: ExpressionType, mask: MaskType, flag: DataFlag, landmarks: Seq[Landmark[_3D]]): Try[Unit] = { val path = landmarksPath(id, expression, mask, flag) path.jfile.getParentFile.mkdirs() val tlms = landmarks.map { lm => TLMSLandmark3D(lm.id, lm.point, visible = true) }.toIndexedSeq val res = TLMSLandmarksIO.write3D(tlms, path.jfile) res match { case Success(_) => setFileAccessMode(path) case _ => } res } def meshPath(id: Person, expression: ExpressionType, mask: MaskType = RAW, flag: DataFlag = Original): Path = { incomingPath / "mesh" / s"${id.id}${expression}${id.raceTag}${mask}$flag.ply" } override def loadMesh(id: Person, expression: ExpressionType) = loadMesh(id, expression, RAW, Original) override def loadMesh(id: Person, expression: ExpressionType, mask: MaskType) = loadMesh(id, expression, mask, Original) override def loadMesh(id: Person, expression: ExpressionType, mask: MaskType, flag: DataFlag): Try[TriangleMesh[_3D]] = { import scalismo.faces.io.MeshIO val path = meshPath(id, expression, mask, flag) MeshIO.read(path.jfile).map(_.shape) } override def loadColoredMesh(id: Person, expression: ExpressionType): Try[ColorNormalMesh3D] = loadColoredMesh(id, expression, RAW, Original) override def loadColoredMesh(id: Person, expression: ExpressionType, mask: MaskType): Try[ColorNormalMesh3D] = loadColoredMesh(id, expression, mask, Original) override def loadColoredMesh(id: Person, expression: ExpressionType, mask: MaskType, flag: DataFlag): Try[ColorNormalMesh3D] = { import scalismo.faces.io.MeshIO val path = meshPath(id, expression, mask, flag) MeshIO.read(path.jfile).map(ocnm => ColorNormalMesh3D(ocnm.shape,ocnm.color.get,ocnm.normals.get)) } override def saveMesh(id: Person, expression: ExpressionType, mesh: TriangleMesh[_3D]): Try[Unit] = saveMesh(id, expression, RAW, Original, mesh) override def saveMesh(id: Person, expression: ExpressionType, mask: MaskType, mesh: TriangleMesh[_3D]): Try[Unit] = saveMesh(id, expression, mask, Original, mesh) override def saveMesh(id: Person, expression: ExpressionType, mask: MaskType, flag: DataFlag, mesh: TriangleMesh[_3D]): Try[Unit] = { import scalismo.faces.io.MeshIO val path = meshPath(id, expression, mask, flag) path.jfile.getParentFile.mkdirs() MeshIO.write(mesh, None, None, path.jfile) setFileAccessMode(path) Success(Unit) } override def saveMesh(id: Person, expression: ExpressionType, mesh: ColorNormalMesh3D): Try[Unit] = saveMesh(id, expression, RAW, Original, mesh) override def saveMesh(id: Person, expression: ExpressionType, mask: MaskType, mesh: ColorNormalMesh3D): Try[Unit] = saveMesh(id, expression, mask, Original, mesh) override def saveMesh(id: Person, expression: ExpressionType, mask: MaskType, flag: DataFlag, mesh: ColorNormalMesh3D): Try[Unit] = { import scalismo.faces.io.MeshIO val path = meshPath(id, expression, mask, flag) path.jfile.getParentFile.mkdirs() MeshIO.write(mesh, path.jfile) setFileAccessMode(path) Success(Unit) } override def ids(expression: ExpressionType): Seq[Person] = { new File(incomingPath.jfile, "mesh").listFiles() .filter(_.getName.endsWith(".ply")) .filter(_.getName.contains(RAW.toString)) .filter(_.getName.contains(expression.toString)) .map(file => BU3DID.fromFilename(file.getName)) .toSeq } } } override def registration: BU3DDataProvider.SurfaceRegistration = { new BU3DDataProvider.SurfaceRegistration { val registrationPath = repositoryRoot / "data" / "registered" registrationPath.jfile.mkdirs() val referencePath = registrationPath / "reference" referencePath.jfile.mkdirs() val modelPath = referencePath / "gpmodels" modelPath.jfile.mkdirs() override def loadPriorModel(expression: ExpressionType): Try[StatisticalMeshModel] = { expression match { case Neutral => StatismoIO.readStatismoMeshModel(new File(modelPath.jfile, "face-model-neutral.h5")) case Sadness => StatismoIO.readStatismoMeshModel(new File(modelPath.jfile, "face-model-sadness.h5")) case Surprise => StatismoIO.readStatismoMeshModel(new File(modelPath.jfile, "face-model-surprise.h5")) case Disgust => StatismoIO.readStatismoMeshModel(new File(modelPath.jfile, "face-model-disgust.h5")) case Fear => StatismoIO.readStatismoMeshModel(new File(modelPath.jfile, "face-model-fear.h5")) case Joy => StatismoIO.readStatismoMeshModel(new File(modelPath.jfile, "face-model-joy.h5")) case Anger => StatismoIO.readStatismoMeshModel(new File(modelPath.jfile, "face-model-anger.h5")) case CoreExpression => StatismoIO.readStatismoMeshModel(new File(modelPath.jfile, "face-model-combined-expressions.h5")) } } override def savePriorModel(model: StatisticalMeshModel, expression: ExpressionType): Try[Unit] = { expression match { case Neutral => StatismoIO.writeStatismoMeshModel(model, new File(modelPath.jfile, "face-model-neutral.h5")) case Sadness => StatismoIO.writeStatismoMeshModel(model, new File(modelPath.jfile, "face-model-sadness.h5")) case Surprise => StatismoIO.writeStatismoMeshModel(model, new File(modelPath.jfile, "face-model-surprise.h5")) case Disgust => StatismoIO.writeStatismoMeshModel(model, new File(modelPath.jfile, "face-model-disgust.h5")) case Fear => StatismoIO.writeStatismoMeshModel(model, new File(modelPath.jfile, "face-model-fear.h5")) case Joy => StatismoIO.writeStatismoMeshModel(model, new File(modelPath.jfile, "face-model-joy.h5")) case Anger => StatismoIO.writeStatismoMeshModel(model, new File(modelPath.jfile, "face-model-anger.h5")) case CoreExpression => StatismoIO.writeStatismoMeshModel(model, new File(modelPath.jfile, "face-model-combined-expressions.h5")) } } def meshPath(id: Person, expression: ExpressionType, mask: MaskType = RAW, flag: DataFlag = Original): Path = { registrationPath / "mesh" / s"${id.id}${expression}${id.raceTag}${mask}${flag}.ply" } override def loadMesh(id: Person, expression: ExpressionType): Try[TriangleMesh[_3D]] = { import scalismo.faces.io.MeshIO val path = meshPath(id, expression) MeshIO.read(path.jfile).map(_.shape) } override def loadMesh(id: Person, expression: ExpressionType, mask: MaskType): Try[TriangleMesh[_3D]] = ??? override def loadMesh(id: Person, expression: ExpressionType, mask: MaskType, flag: DataFlag): Try[TriangleMesh[_3D]] = ??? override def loadColoredMesh(id: Person, expression: ExpressionType): Try[ColorNormalMesh3D] = ??? override def loadColoredMesh(id: Person, expression: ExpressionType, mask: MaskType): Try[ColorNormalMesh3D] = ??? override def loadColoredMesh(id: Person, expression: ExpressionType, mask: MaskType, flag: DataFlag): Try[ColorNormalMesh3D] = ??? override def saveMesh(id: Person, expression: ExpressionType, mesh: TriangleMesh[_3D]): Try[Unit] = { import scalismo.faces.io.MeshIO val path = meshPath(id, expression) MeshIO.write(mesh, None, None, path.jfile) setFileAccessMode(path) Success(Unit) } override def saveMesh(id: Person, expression: ExpressionType, mask: MaskType, mesh: TriangleMesh[_3D]): Try[Unit] = ??? override def saveMesh(id: Person, expression: ExpressionType, mask: MaskType, flag: DataFlag, mesh: TriangleMesh[_3D]): Try[Unit] = ??? override def saveMesh(id: Person, expression: ExpressionType, mesh: ColorNormalMesh3D): Try[Unit] = ??? override def saveMesh(id: Person, expression: ExpressionType, mask: MaskType, mesh: ColorNormalMesh3D): Try[Unit] = ??? override def saveMesh(id: Person, expression: ExpressionType, mask: MaskType, flag: DataFlag, mesh: ColorNormalMesh3D): Try[Unit] = ??? override def ids(expression: ExpressionType): Seq[Person] = { new File(registrationPath.jfile, "mesh").listFiles() .filter(_.getName.endsWith("ply")) .filter(_.getName.contains(expression.toString)) .map(file => BU3DID.fromFilename(file.getName)) .toSeq } } } override def model : BU3DDataProvider.ModelBuilding = BU3DModelBuilding object BU3DModelBuilding extends BU3DDataProvider.ModelBuilding { val modelBuildingPath = repositoryRoot / "data" / "modelbuilding" modelBuildingPath.jfile.mkdirs() val modelDirectoryPath = modelBuildingPath / "model" modelDirectoryPath.jfile.mkdirs() val colorExtractdMeshPath = modelBuildingPath / "mesh" colorExtractdMeshPath.jfile.mkdirs() def meshPath(id: Person, expression: ExpressionType, mask: MaskType = RAW, flag: DataFlag = Original): Path = { colorExtractdMeshPath / s"${id.id}${expression}${id.raceTag}${mask}$flag.ply" } override def loadColoredMesh(id: Person, expression: ExpressionType): Try[ColorNormalMesh3D] = { loadColoredMesh(id,expression,RAW,Original) } def loadColoredMesh(id: Person, expression: ExpressionType, mask: MaskType = RAW, flag: DataFlag = Original): Try[ColorNormalMesh3D] = { import scalismo.faces.io.MeshIO val path = meshPath(id, expression, mask, flag) MeshIO.read(path.jfile).map(_.colorNormalMesh3D.get) } override def saveColoredMesh(id: Person, expression: ExpressionType, mesh: ColorNormalMesh3D): Try[Unit] = { saveColoredMesh(id,expression,mesh,RAW,Original) } def saveColoredMesh(id: Person, expression: ExpressionType, mesh: ColorNormalMesh3D, mask: MaskType = RAW, flag: DataFlag = Original): Try[Unit] = { import scalismo.faces.io.MeshIO val path = meshPath(id, expression, mask, flag) path.jfile.getParentFile.mkdirs MeshIO.write(mesh, path.jfile) setFileAccessMode(path) Success(Unit) } def modelPath(mask: MaskType): Path = { modelDirectoryPath / s"bu3d_pami17${mask}.h5" // @todo think about registration identifier in name } override def saveModel(mask: MaskType, momo: MoMo): Try[Unit] = { val path = modelPath(mask) path.jfile.getParentFile.mkdirs val res = MoMoIO.write(momo, path.jfile, "") res match { case Success(_) => setFileAccessMode(path) case _ => } res } override def loadModel(mask: MaskType): Try[MoMo] = { MoMoIO.read(modelPath(mask).jfile, "") } } override def fitting: BU3DDataProvider.Fitting = { new BU3DDataProvider.Fitting { } } override def loadMeshMask(from: String, to: String): Try[BinaryMask] = { BinaryMask.load(Source.fromFile(new File(repositoryRoot / "data" / "incoming" / "reference" / "masks" / from+"_TO_"+to+".mask"))) } } ================================================ FILE: src/main/scala/ch/unibas/cs/gravis/facepipeline/DataProvider.scala ================================================ /* * Copyright University of Basel, Graphics and Vision Research Group * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ package ch.unibas.cs.gravis.facepipeline import registration.modelbuilding.FaceMask import scalismo.faces.mesh.{BinaryMask, ColorNormalMesh3D} import scalismo.faces.momo.MoMo import scalismo.geometry.{Landmark, _3D} import scalismo.mesh.TriangleMesh import scalismo.statisticalmodel.StatisticalMeshModel import scala.reflect.io.Path import scala.util.Try trait ExpressionType { override def toString : String } trait MaskType { override def toString: String } trait DataFlag { override def toString: String } trait DataProvider { trait Person { def id: String def raceTag: String } trait WithMesh { def loadMesh(id: Person, expression: ExpressionType): Try[TriangleMesh[_3D]] def loadMesh(id: Person, expression: ExpressionType, mask: MaskType): Try[TriangleMesh[_3D]] def loadMesh(id: Person, expression: ExpressionType, mask: MaskType, flag: DataFlag): Try[TriangleMesh[_3D]] def loadColoredMesh(id: Person, expression: ExpressionType): Try[ColorNormalMesh3D] def loadColoredMesh(id: Person, expression: ExpressionType, mask: MaskType): Try[ColorNormalMesh3D] def loadColoredMesh(id: Person, expression: ExpressionType, mask: MaskType, flag: DataFlag): Try[ColorNormalMesh3D] def saveMesh(id: Person, expression: ExpressionType, mesh: TriangleMesh[_3D]): Try[Unit] def saveMesh(id: Person, expression: ExpressionType, mask: MaskType, mesh: TriangleMesh[_3D]): Try[Unit] def saveMesh(id: Person, expression: ExpressionType, mask: MaskType, flag: DataFlag, mesh: TriangleMesh[_3D]): Try[Unit] def saveMesh(id: Person, expression: ExpressionType, mesh: ColorNormalMesh3D): Try[Unit] def saveMesh(id: Person, expression: ExpressionType, mask: MaskType, mesh: ColorNormalMesh3D): Try[Unit] def saveMesh(id: Person, expression: ExpressionType, mask: MaskType, flag: DataFlag, mesh: ColorNormalMesh3D): Try[Unit] } trait WithLandmarks { def loadLandmarks(id: Person, expression: ExpressionType): Try[Seq[Landmark[_3D]]] def loadLandmarks(id: Person, expression: ExpressionType, mask: MaskType): Try[Seq[Landmark[_3D]]] def loadLandmarks(id: Person, expression: ExpressionType, mask: MaskType, flag: DataFlag): Try[Seq[Landmark[_3D]]] def saveLandmarks(id: Person, expression: ExpressionType, landmarks: Seq[Landmark[_3D]]): Try[Unit] def saveLandmarks(id: Person, expression: ExpressionType, mask: MaskType, landmarks: Seq[Landmark[_3D]]): Try[Unit] def saveLandmarks(id: Person, expression: ExpressionType, mask: MaskType, flag: DataFlag, landmarks: Seq[Landmark[_3D]]): Try[Unit] } trait WithLineLandmarks { def loadLineLandmarks(id: Person, expression: ExpressionType): Try[Seq[Landmark[_3D]]] def saveLineLandmarks(id: Person, expression: ExpressionType): Try[Seq[Landmark[_3D]]] } trait WithIds { def ids(expression: ExpressionType): Seq[Person] } trait Reference { def loadMesh(expression: ExpressionType): Try[TriangleMesh[_3D]] def loadFaceMask(): Try[FaceMask] def loadLandmarks(expression: ExpressionType): Try[Seq[Landmark[_3D]]] def saveLandmarks(expression: ExpressionType, landmarks : Seq[Landmark[_3D]]): Try[Unit] def loadLineLandmarks(expression: ExpressionType): Try[Seq[Landmark[_3D]]] } trait Incoming extends WithIds with WithMesh with WithLandmarks { def reference: Reference } trait SurfaceRegistration extends WithIds with WithMesh { def loadPriorModel(expression: ExpressionType): Try[StatisticalMeshModel] def savePriorModel(model: StatisticalMeshModel, expressionType: ExpressionType): Try[Unit] } trait ModelBuilding { def loadModel( mask: MaskType ) : Try[MoMo] def saveModel( mask: MaskType, momo: MoMo ) : Try[Unit] def loadColoredMesh(id: Person, expression: ExpressionType): Try[ColorNormalMesh3D] def saveColoredMesh(id: Person, expression: ExpressionType, mesh: ColorNormalMesh3D): Try[Unit] } trait Fitting {} def Neutral: ExpressionType def repositoryRoot: Path def incoming: Incoming def registration: SurfaceRegistration def model: ModelBuilding def fitting: Fitting def expressions: Seq[ExpressionType] def masks: Seq[MaskType] def loadMeshMask(from: String, to: String): Try[BinaryMask] def personFromFilename(filename: String): Person } ================================================ FILE: src/main/scala/ch/unibas/cs/gravis/facepipeline/PipelineStep.scala ================================================ /* * Copyright University of Basel, Graphics and Vision Research Group * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ package ch.unibas.cs.gravis.facepipeline trait PipelineStep { def run() : Unit } ================================================ FILE: src/main/scala/fitting/StandardFitScript.scala ================================================ /* * Copyright University of Basel, Graphics and Vision Research Group * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ package fitting import java.io.File import scalismo.faces.color.{RGB, RGBA} import scalismo.faces.deluminate.SphericalHarmonicsOptimizer import scalismo.faces.image.PixelImage import scalismo.faces.io.{PixelImageIO, RenderParameterIO, TLMSLandmarksIO} import scalismo.faces.mesh.MeshSurfaceSampling import scalismo.faces.parameters.RenderParameter import scalismo.faces.sampling.face.evaluators.PixelEvaluators._ import scalismo.faces.sampling.face.evaluators.PointEvaluators.IsotropicGaussianPointEvaluator import scalismo.faces.sampling.face.evaluators.PriorEvaluators.{GaussianShapePrior, GaussianTexturePrior} import scalismo.faces.sampling.face.evaluators._ import scalismo.faces.sampling.face.loggers._ import scalismo.faces.sampling.face.proposals.ImageCenteredProposal.implicits._ import scalismo.faces.sampling.face.proposals.ParameterProposals.implicits._ import scalismo.faces.sampling.face.proposals.SphericalHarmonicsLightProposals._ import scalismo.faces.sampling.face.proposals._ import scalismo.faces.sampling.face.{MoMoRenderer, ParametricLandmarksRenderer, ParametricModel} import scalismo.geometry.{Vector, Vector3D, _2D} import scalismo.sampling.algorithms.MetropolisHastings import scalismo.sampling.evaluators.ProductEvaluator import scalismo.sampling.loggers.ChainStateLogger.implicits._ import scalismo.sampling.loggers.ChainStateLoggerContainer.implicits._ import scalismo.sampling.loggers.{BestSampleLogger} import scalismo.sampling.proposals.MixtureProposal.implicits._ import scalismo.sampling.proposals.{MetropolisFilterProposal, MixtureProposal} import scalismo.sampling.{ProposalGenerator, TransitionProbability} import scalismo.utils.Random /* This Fitscript with its evaluators and the proposal distribution follows closely the proposed setting of: Markov Chain Monte Carlo for Automated Face Image Analysis Sandro Sch�nborn, Bernhard Egger, Andreas Morel-Forster and Thomas Vetter International Journal of Computer Vision 123(2), 160-183 , June 2017 DOI: http://dx.doi.org/10.1007/s11263-016-0967-5 To understand the concepts behind the fitscript and the underlying methods there is a tutorial on: http://gravis.dmi.unibas.ch/pmm/ */ object StandardFitScript { /* Collection of all pose related proposals */ def defaultPoseProposal(lmRenderer: ParametricLandmarksRenderer)(implicit rnd: Random): ProposalGenerator[RenderParameter] with TransitionProbability[RenderParameter] = { import MixtureProposal.implicits._ val yawProposalC = GaussianRotationProposal(Vector3D.unitY, 0.75f) val yawProposalI = GaussianRotationProposal(Vector3D.unitY, 0.10f) val yawProposalF = GaussianRotationProposal(Vector3D.unitY, 0.01f) val rotationYaw = MixtureProposal(0.1 *: yawProposalC + 0.4 *: yawProposalI + 0.5 *: yawProposalF) val pitchProposalC = GaussianRotationProposal(Vector3D.unitX, 0.75f) val pitchProposalI = GaussianRotationProposal(Vector3D.unitX, 0.10f) val pitchProposalF = GaussianRotationProposal(Vector3D.unitX, 0.01f) val rotationPitch = MixtureProposal(0.1 *: pitchProposalC + 0.4 *: pitchProposalI + 0.5 *: pitchProposalF) val rollProposalC = GaussianRotationProposal(Vector3D.unitZ, 0.75f) val rollProposalI = GaussianRotationProposal(Vector3D.unitZ, 0.10f) val rollProposalF = GaussianRotationProposal(Vector3D.unitZ, 0.01f) val rotationRoll = MixtureProposal(0.1 *: rollProposalC + 0.4 *: rollProposalI + 0.5 *: rollProposalF) val rotationProposal = MixtureProposal(0.5 *: rotationYaw + 0.3 *: rotationPitch + 0.2 *: rotationRoll).toParameterProposal val translationC = GaussianTranslationProposal(Vector(300f, 300f)).toParameterProposal val translationF = GaussianTranslationProposal(Vector(50f, 50f)).toParameterProposal val translationHF = GaussianTranslationProposal(Vector(10f, 10f)).toParameterProposal val translationProposal = MixtureProposal(0.2 *: translationC + 0.2 *: translationF + 0.6 *: translationHF) val distanceProposalC = GaussianDistanceProposal(500f, compensateScaling = true).toParameterProposal val distanceProposalF = GaussianDistanceProposal(50f, compensateScaling = true).toParameterProposal val distanceProposalHF = GaussianDistanceProposal(5f, compensateScaling = true).toParameterProposal val distanceProposal = MixtureProposal(0.2 *: distanceProposalC + 0.6 *: distanceProposalF + 0.2 *: distanceProposalHF) val scalingProposalC = GaussianScalingProposal(0.15f).toParameterProposal val scalingProposalF = GaussianScalingProposal(0.05f).toParameterProposal val scalingProposalHF = GaussianScalingProposal(0.01f).toParameterProposal val scalingProposal = MixtureProposal(0.2 *: scalingProposalC + 0.6 *: scalingProposalF + 0.2 *: scalingProposalHF) val poseMovingNoTransProposal = MixtureProposal(rotationProposal + distanceProposal + scalingProposal) val centerREyeProposal = poseMovingNoTransProposal.centeredAt("right.eye.corner_outer", lmRenderer).get val centerLEyeProposal = poseMovingNoTransProposal.centeredAt("left.eye.corner_outer", lmRenderer).get val centerRLipsProposal = poseMovingNoTransProposal.centeredAt("right.lips.corner", lmRenderer).get val centerLLipsProposal = poseMovingNoTransProposal.centeredAt("left.lips.corner", lmRenderer).get MixtureProposal(centerREyeProposal + centerLEyeProposal + centerRLipsProposal + centerLLipsProposal + 0.2 *: translationProposal) } /* Collection of all illumination related proposals */ def defaultIlluminationProposal(modelRenderer: ParametricModel, target: PixelImage[RGBA])(implicit rnd: Random): ProposalGenerator[RenderParameter] with TransitionProbability[RenderParameter] = { val shOpt = SphericalHarmonicsOptimizer(modelRenderer, target) val shOptimizerProposal = SHLightSolverProposal(shOpt, MeshSurfaceSampling.sampleUniformlyOnSurface(100)) val lightSHPert = SHLightPerturbationProposal(0.001f, fixIntensity = true) val lightSHIntensity = SHLightIntensityProposal(0.1f) val lightSHBandMixter = SHLightBandEnergyMixer(0.1f) val lightSHSpatial = SHLightSpatialPerturbation(0.05f) val lightSHColor = SHLightColorProposal(0.01f) MixtureProposal((5f / 6f) *: MixtureProposal(lightSHSpatial + lightSHBandMixter + lightSHIntensity + lightSHPert + lightSHColor).toParameterProposal + (1f / 6f) *: shOptimizerProposal) } /* Collection of all statistical model (shape, texture) related proposals */ def neutralMorphableModelProposal(implicit rnd: Random): ProposalGenerator[RenderParameter] with TransitionProbability[RenderParameter] = { val shapeC = GaussianMoMoShapeProposal(0.2f) val shapeF = GaussianMoMoShapeProposal(0.1f) val shapeHF = GaussianMoMoShapeProposal(0.025f) val shapeScaleProposal = GaussianMoMoShapeCaricatureProposal(0.2f) val shapeProposal = MixtureProposal(0.1f *: shapeC + 0.5f *: shapeF + 0.2f *: shapeHF + 0.2f *: shapeScaleProposal).toParameterProposal val textureC = GaussianMoMoColorProposal(0.2f) val textureF = GaussianMoMoColorProposal(0.1f) val textureHF = GaussianMoMoColorProposal(0.025f) val textureScale = GaussianMoMoColorCaricatureProposal(0.2f) val textureProposal = MixtureProposal(0.1f *: textureC + 0.5f *: textureF + 0.2 *: textureHF + 0.2f *: textureScale).toParameterProposal MixtureProposal(shapeProposal + textureProposal ) } /* Collection of all statistical model (shape, texture, expression) related proposals */ def defaultMorphableModelProposal(implicit rnd: Random): ProposalGenerator[RenderParameter] with TransitionProbability[RenderParameter] = { val expressionC = GaussianMoMoExpressionProposal(0.2f) val expressionF = GaussianMoMoExpressionProposal(0.1f) val expressionHF = GaussianMoMoExpressionProposal(0.025f) val expressionScaleProposal = GaussianMoMoExpressionCaricatureProposal(0.2f) val expressionProposal = MixtureProposal(0.1f *: expressionC + 0.5f *: expressionF + 0.2f *: expressionHF + 0.2f *: expressionScaleProposal).toParameterProposal MixtureProposal(neutralMorphableModelProposal + expressionProposal) } /* Collection of all color transform proposals */ def defaultColorProposal(implicit rnd: Random): ProposalGenerator[RenderParameter] with TransitionProbability[RenderParameter] = { val colorC = GaussianColorProposal(RGB(0.01f, 0.01f, 0.01f), 0.01f, RGB(1e-4f, 1e-4f, 1e-4f)) val colorF = GaussianColorProposal(RGB(0.001f, 0.001f, 0.001f), 0.01f, RGB(1e-4f, 1e-4f, 1e-4f)) val colorHF = GaussianColorProposal(RGB(0.0005f, 0.0005f, 0.0005f), 0.01f, RGB(1e-4f, 1e-4f, 1e-4f)) MixtureProposal(0.2f *: colorC + 0.6f *: colorF + 0.2f *: colorHF).toParameterProposal } def fit(targetFn : String, lmFn: String, outputDir: String, modelRenderer: MoMoRenderer, expression: Boolean = true)(implicit rnd: Random):RenderParameter = { val target = PixelImageIO.read[RGBA](new File(targetFn)).get val targetLM = TLMSLandmarksIO.read2D(new File(lmFn)).get.filter(lm => lm.visible) PixelImageIO.write(target, new File(s"$outputDir/target.png")).get val init: RenderParameter = RenderParameter.defaultSquare.fitToImageSize(target.width, target.height) val sdev = 0.043f /* Foreground Evaluator */ val pixEval = IsotropicGaussianPixelEvaluator(sdev) /* Background Evaluator */ val histBGEval = HistogramRGB.fromImageRGBA(target, 25) /* Pixel Evaluator */ val imgEval = IndependentPixelEvaluator(pixEval, histBGEval) /* Prior Evaluator */ val priorEval = ProductEvaluator(GaussianShapePrior(0, 1), GaussianTexturePrior(0, 1)) /* Image Evaluator */ val allEval = ImageRendererEvaluator(modelRenderer, imgEval.toDistributionEvaluator(target)) /* Landmarks Evaluator */ val pointEval = IsotropicGaussianPointEvaluator[_2D](4.0) //lm click uncertainty in pixel! -> should be related to image/face size val landmarksEval = LandmarkPointEvaluator(targetLM, pointEval, modelRenderer) //logging val imageLogger = ImageRenderLogger(modelRenderer, new File(s"$outputDir/"), "mc-").withBackground(target) // Metropolis logger val printLogger = PrintLogger[RenderParameter](Console.out, "").verbose val mhLogger = printLogger // keep track of best sample val bestFileLogger = ParametersFileBestLogger(allEval, new File(s"$outputDir/fit-best.rps")) val bestSampleLogger = BestSampleLogger(allEval) val parametersLogger = ParametersFileLogger(new File(s"$outputDir/"), "mc-") val fitLogger = bestFileLogger :+ bestSampleLogger // pose proposal val totalPose = defaultPoseProposal(modelRenderer) //light proposals val lightProposal = defaultIlluminationProposal(modelRenderer, target) //color proposals val colorProposal = defaultColorProposal //Morphable Model proposals val momoProposal = if(expression) defaultMorphableModelProposal else neutralMorphableModelProposal // full proposal filtered by the landmark and prior Evaluator val proposal = MetropolisFilterProposal(MetropolisFilterProposal(MixtureProposal(totalPose + colorProposal + 3f*:momoProposal + 2f *: lightProposal), landmarksEval), priorEval) //pose and image chains val imageFitter = MetropolisHastings(proposal, allEval) val poseFitter = MetropolisHastings(totalPose, landmarksEval) println("everyting setup. starting fitter ...") //landmark chain for initialisation val initDefault: RenderParameter = RenderParameter.defaultSquare.fitToImageSize(target.width, target.height) val init10 = initDefault.withMoMo(init.momo.withNumberOfCoefficients(50, 50, 5)) val initLMSamples: IndexedSeq[RenderParameter] = poseFitter.iterator(init10, mhLogger).take(5000).toIndexedSeq val lmScores = initLMSamples.map(rps => (landmarksEval.logValue(rps), rps)) val bestLM = lmScores.maxBy(_._1)._2 RenderParameterIO.write(bestLM, new File(s"$outputDir/fitter-lminit.rps")).get val imgLM = modelRenderer.renderImage(bestLM) PixelImageIO.write(imgLM, new File(s"$outputDir/fitter-lminit.png")).get def printer(sample: RenderParameter): RenderParameter = { println(s"${sample.momo.shape} ${sample.momo.color} ${sample.momo.expression}") sample } // image chain, fitting val fitsamples = imageFitter.iterator(bestLM, mhLogger).loggedWith(fitLogger).take(10000).toIndexedSeq val best = bestSampleLogger.currentBestSample().get val imgBest = modelRenderer.renderImage(best) PixelImageIO.write(imgBest, new File(s"$outputDir/fitter-best.png")).get best } } ================================================ FILE: src/main/scala/fitting/experiments/QualitativeLFW.scala ================================================ /* * Copyright University of Basel, Graphics and Vision Research Group * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ package fitting.experiments import java.io.File import ch.unibas.cs.gravis.facepipeline.BU3DDataProvider import fitting.StandardFitScript import scalismo.faces.color.RGBA import scalismo.faces.io.MoMoIO import scalismo.faces.momo.MoMo import scalismo.faces.sampling.face.MoMoRenderer import scalismo.utils.Random import scala.reflect.io.Path object QualitativeLFW extends App{ scalismo.initialize() val seed = 1986L implicit val rnd = new Random(seed) def fitModel(model:MoMo, modelName: String) = { val targetsPath = BU3DDataProvider.repositoryRoot + "/recognition-experiment/fit-lfw-qualitative/lfwSelection/" val outPath = BU3DDataProvider.repositoryRoot + "/recognition-experiment/fit-lfw-qualitative/lfwResults/" + modelName + "/" val files = new File(targetsPath).listFiles.filter(_.getName.endsWith(".png")) val listTarget = files.map(p => p.getName.substring(0, p.getName.length - 4)).toList listTarget.foreach{ targetName => val outPathTarget = outPath + targetName + "/" if (!Path(outPathTarget).exists) { try { Path(outPathTarget).createDirectory(failIfExists = false) val renderer = MoMoRenderer(model, RGBA.BlackTransparent).cached(5) val targetFn = targetsPath + targetName + ".png" val targetLM = targetsPath + targetName + "_face0.tlms" StandardFitScript.fit(targetFn, targetLM, outPathTarget, renderer) } } } } val bfm = MoMoIO.read(new File( BU3DDataProvider.repositoryRoot + "/data/modelbuilding/model/model2017-1_face12_nomouth.h5")).get val bfmOld = MoMoIO.read(new File(BU3DDataProvider.repositoryRoot + "/model2009-face12.h5")).get val bu3d = MoMoIO.read(new File(BU3DDataProvider.repositoryRoot + "/data/modelbuilding/model/bu3d-face12_nomouth.h5")).get fitModel(bfm, "bfm") fitModel(bfmOld, "bfmOld") fitModel(bu3d, "bu3d") } ================================================ FILE: src/main/scala/fitting/experiments/RecognitionMultiPIE.scala ================================================ /* * Copyright University of Basel, Graphics and Vision Research Group * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ package fitting.experiments import java.io.File import ch.unibas.cs.gravis.facepipeline.BU3DDataProvider import fitting.StandardFitScript import scalismo.faces.color.RGBA import scalismo.faces.io.{MoMoIO, RenderParameterIO} import scalismo.faces.momo.MoMo import scalismo.faces.sampling.face.MoMoRenderer import scalismo.utils.Random import scala.reflect.io.Path // script to fit multiPie Neutral Samples with landmarks object RecognitionMultiPiePose extends App{ scalismo.initialize() val seed = 1986L implicit val rnd = new Random(seed) def fitModel(model:MoMo, modelName: String) = { val targetsPath = BU3DDataProvider.repositoryRoot + "/recognition-experiment" val outPath = targetsPath + "/results/" + modelName + "/" val files = new File(targetsPath + "/originals/").listFiles.filter(_.getName.endsWith(".png")) val listTarget = files.map(p => p.getName.substring(0, p.getName.length - 4)).toList listTarget.foreach(targetName => { val outPathTarget = outPath + targetName + "/" if (!Path(outPathTarget).exists) { try { Path(outPathTarget).createDirectory(failIfExists = false) val renderer = MoMoRenderer(model, RGBA.BlackTransparent).cached(5) val targetFn = targetsPath + "/originals/" + targetName + ".png" val targetLM = targetsPath + "/landmarks/" + targetName + "_face0.tlms" StandardFitScript.fit(targetFn, targetLM, outPathTarget, renderer, false) } } }) } val bfm = MoMoIO.read(new File(BU3DDataProvider.repositoryRoot + "data/modelbuilding/model/model2017-1_face12_nomouth.h5")).get.neutralModel val bfmOld = MoMoIO.read(new File(BU3DDataProvider.repositoryRoot + "/export/faces/model/bfm2009/model2009-face12.h5")).get.neutralModel val bu3d = MoMoIO.read(new File(BU3DDataProvider.repositoryRoot + "/export/faces/projects/pami-ppm2017/basel-face-pipeline/data/modelbuilding/model/bu3d-face12_nomouth.h5")).get.neutralModel fitModel(bfm, "bfm") fitModel(bfmOld, "bfmOld") fitModel(bu3d, "bu3d") } // script to fit multiPie Expression Samples with landmarks object RecognitionMultiPieExpression extends App{ scalismo.initialize() val seed = 1986L implicit val rnd = new Random(seed) def fitModel(model:MoMo, modelName: String) = { val targetsPath = "/export/faces/projects/pami-ppm2017/experiments/fit-multipie-recognition/multipie/" val outPath = targetsPath + "/results/" + modelName + "/" val files = new File(targetsPath + "/originalsExpressionsNotForPublishing").listFiles.filter(_.getName.endsWith(".png")) val listTarget = files.map(p => p.getName.substring(0, p.getName.length - 4)).toList listTarget.foreach(targetName => { val outPathTarget = outPath + targetName + "/" if (!Path(outPathTarget).exists) { try { Path(outPathTarget).createDirectory(failIfExists = false) val renderer = MoMoRenderer(model, RGBA.BlackTransparent).cached(5) val targetFn = targetsPath + "/originalsExpressionsNotForPublishing/" + targetName + ".png" val targetLM = targetsPath + "landmarksExpressions/" + targetName + "_face0.tlms" StandardFitScript.fit(targetFn, targetLM, outPathTarget, renderer) } } }) } val bu3dEx = MoMoIO.read(new File(BU3DDataProvider.repositoryRoot + "/data/modelbuilding/model/bu3d-face12_nomouth.h5")).get val bfmEx = MoMoIO.read(new File(BU3DDataProvider.repositoryRoot + "/data/modelbuilding/model/model2017-1_face12_nomouth.h5")).get fitModel(bfmEx, "bfmEx") fitModel(bu3dEx, "bu3dEx") } // Script to calculate recognition results over pose fitted multipie data object RecognitionEvaluation extends App { case class Fit( id: String, pose: String, coeffs: IndexedSeq[Double]) case class Match( id: String, similarity: Double) val models = IndexedSeq("bfmOld", "bu3d", "bfm") val resultsPath = BU3DDataProvider.repositoryRoot + "/recognition-experiment/results/" models.foreach { model => val resultPath = resultsPath + model + "/" val files = new File(resultPath).listFiles.filter(_.isDirectory).toIndexedSeq.sortBy(_.getAbsoluteFile) val allFits = files.map { f => val name = f.getName val id = name.substring(0, 3) val pose = name.substring(10, 13) val rps = RenderParameterIO.read(new File(resultPath + name + "/fit-best.rps")).get val coeffs = rps.momo.color ++ rps.momo.shape Fit(id, pose, coeffs) } val gallery = allFits.filter(fit => fit.pose == "051") val listOfPoses = IndexedSeq("051", "140", "130", "080") val listOfExperiments = listOfPoses.map { queryPose => allFits.filter(fit => fit.pose == queryPose) } val queriesWithSimilarities = listOfExperiments.map { queriesInExperiment => queriesInExperiment.map { query => val similaritiesForQuery = gallery.map { subject => Match(subject.id, cosineAngle(query.coeffs, subject.coeffs)) } (query.id, similaritiesForQuery) } } val correctMatchesPerExperiment: IndexedSeq[Double] = queriesWithSimilarities.map { experiment => val correctMatches = experiment.map { case (query_id, similarities) => val bestMatch = similarities.maxBy(m => m.similarity) //println(query_id, bestMatch) if (bestMatch.id == query_id) 1.0 else 0.0 }.sum correctMatches / experiment.length } println(model + correctMatchesPerExperiment) } def cosineAngle(aa: IndexedSeq[Double], bb: IndexedSeq[Double]): Double = { import breeze.linalg._ val a = DenseVector(aa.toArray) val b = DenseVector(bb.toArray) (a dot b) / (norm(a) * norm(b)) } } // Script to calculate recognition results over expression on fitted multipie data object RecognitionEvaluationEx extends App { case class Fit( id: String, pose: String, coeffs: IndexedSeq[Double]) case class Match( id: String, similarity: Double) val models = IndexedSeq("bfmEx", "bu3dEx") val resultsPath = BU3DDataProvider.repositoryRoot + "/recognition-experiment/results/" models.foreach { model => val resultPath = resultsPath + model + "/" val files = new File(resultPath).listFiles.filter(_.isDirectory).toIndexedSeq.sortBy(_.getAbsoluteFile) val allFits = files.map { f => val name = f.getName val id = name.substring(0, 3) val expression = name.substring(7, 9) val rps = RenderParameterIO.read(new File(resultPath + name + "/fit-best.rps")).get val coeffs = rps.momo.color ++ rps.momo.shape Fit(id, expression, coeffs) } val gallery = allFits.filter(fit => fit.pose == "01") val listOfPoses = IndexedSeq("01","02") val listOfExperiments = listOfPoses.map { queryPose => allFits.filter(fit => fit.pose == queryPose) } val queriesWithSimilarities = listOfExperiments.map { queriesInExperiment => queriesInExperiment.map { query => val similaritiesForQuery = gallery.map { subject => Match(subject.id, cosineAngle(query.coeffs, subject.coeffs)) } (query.id, similaritiesForQuery) } } val correctMatchesPerExperiment: IndexedSeq[Double] = queriesWithSimilarities.map { experiment => val correctMatches = experiment.map { case (query_id, similarities) => val bestMatch = similarities.maxBy(m => m.similarity) // println(query_id, bestMatch) if (bestMatch.id == query_id) 1.0 else 0.0 }.sum correctMatches / experiment.length } println(model + correctMatchesPerExperiment) } def cosineAngle(aa: IndexedSeq[Double], bb: IndexedSeq[Double]): Double = { import breeze.linalg._ val a = DenseVector(aa.toArray) val b = DenseVector(bb.toArray) (a dot b) / (norm(a) * norm(b)) } } ================================================ FILE: src/main/scala/modelbuilding/ModelBuilding.scala ================================================ /* * Copyright University of Basel, Graphics and Vision Research Group * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ package modelbuilding import breeze.linalg.DenseMatrix import ch.unibas.cs.gravis.facepipeline.{BU3DDataProvider, DataProvider, ExpressionType, PipelineStep} import scalismo.common._ import scalismo.faces.color.{RGB, RGBA} import scalismo.faces.io.MoMoIO import scalismo.faces.mesh.{ColorNormalMesh3D, VertexColorMesh3D} import scalismo.faces.momo.MoMo.NeutralWithExpression import scalismo.faces.momo.{MoMo, PancakeDLRGP} import scalismo.faces.render.Transform3D import scalismo.geometry.{Point, Vector, _3D} import scalismo.kernels.{DiagonalKernel, GaussianKernel, MatrixValuedPDKernel} import scalismo.mesh.{SurfacePointProperty, TriangleMesh, TriangleMesh3D} import scalismo.numerics.{UniformMeshSampler3D} import scalismo.registration.{LandmarkRegistration, RigidTransformation} import scalismo.statisticalmodel.{GaussianProcess, LowRankGaussianProcess} import scala.util.{Success, Try} object ModelBuilding { def main(args: Array[String]) { scalismo.initialize() ModelBuilding(BU3DDataProvider).run() } } case class ModelBuilding(dataProvider: DataProvider) extends PipelineStep { override def run() { createMeshesWithVertexColor() // buildMoMoExpress() buildMoMoExpress("face12_nomouth") } /** * Extracts color for all registration results from the input meshes. */ def createMeshesWithVertexColor(): Unit = { println("Extracting color from meshes ...") val expressions = dataProvider.expressions expressions.zipWithIndex.flatMap { case (exp,idx) => println(s"... processing expression ${exp} (${idx+1}/${expressions.size})") val ids = dataProvider.registration.ids(exp) ids.zipWithIndex.map { case(id,idx) => println(s"... ... processing id ${id} (${idx+1}/${ids.size})") val registeredShape: TriangleMesh[_3D] = dataProvider.registration.loadMesh(id, exp).get val coloredMesh: ColorNormalMesh3D = dataProvider.incoming.loadColoredMesh(id, exp).get val vertexColor = extractVertexColor(registeredShape, coloredMesh) val registeredShapeWithVertexColor = ColorNormalMesh3D(registeredShape, vertexColor, registeredShape.vertexNormals) dataProvider.model.saveColoredMesh(id, exp, registeredShapeWithVertexColor).get } } } /** * Extracts per vertex color for the registered mesh from the input mesh. The correspondence for each * vertex is sought after along the normal. * * @param registeredMesh Registration result without color. * @param colorMesh Input mesh with color. * @return Registeration result with color. */ def extractVertexColor(registeredMesh: TriangleMesh3D, colorMesh: ColorNormalMesh3D): SurfacePointProperty[RGBA] = { val pointsReg = registeredMesh.pointSet.points.toIndexedSeq val normalsReg = registeredMesh.vertexNormals.pointData val shapeFromColorMesh = colorMesh.shape val meshOperations = shapeFromColorMesh.operations val colors = pointsReg.zip(normalsReg).map { case (point, normal) => val intersections = meshOperations.getIntersectionPointsOnSurface(point, normal) val sortedIntersections = intersections.map { i => val pointI = shapeFromColorMesh.position(i._1, i._2) val dist = (pointI - point).norm (i, dist) }.sortWith((a, b) => a._2 < b._2) if (sortedIntersections.nonEmpty && sortedIntersections.head._2 < 2.0) { val i = sortedIntersections.head._1 colorMesh.color(i._1, i._2) } else { RGBA(0.9, 0.8, 0.1, 0.0) } } new SurfacePointProperty[RGBA](registeredMesh.triangulation, colors) } /** * Build a model from meshes with vertex color. * This step assumes that the registration was performed using the "bfm_nomouth" masked reference. */ def buildMoMoExpress(maskType: String = "bfm_nomouth"): Unit = { println("Building model from meshes with vertex color...") val mask = { if ( maskType != "bfm_nomouth" ) Some(dataProvider.loadMeshMask("bfm_nomouth", "face12_nomouth").get) else None } val reference = { val originalRef = dataProvider.registration.loadPriorModel(dataProvider.Neutral).get.referenceMesh if ( mask.isDefined ) originalRef.operations.maskPoints(mask.get).transformedMesh else originalRef } val ids = dataProvider.registration.ids(dataProvider.Neutral) val otherExpressions = dataProvider.expressions.filter(exp => exp != dataProvider.Neutral) buildModel(ids, otherExpressions, reference) def buildModel(ids: Seq[dataProvider.Person], expressions: Seq[ExpressionType], reference: TriangleMesh3D) = { val data: Seq[Try[(DiscreteField[_3D, RGBA], VertexColorMesh3D, Seq[NeutralWithExpression])]] = ids.zipWithIndex.map { case (id, idx) => println(s"... loading data for ${id.id} (${idx + 1}/${ids.size})") prepareData(reference, expressions, id) } val neutralMeshes: Seq[VertexColorMesh3D] = data.collect({ case Success(e) => e._2 }) val neutralWithExpressions: Seq[NeutralWithExpression] = data.collect({ case Success(e) => e._3 }).flatten println(".. data loaded ...") val momo = MoMo.buildFromRegisteredSamples( reference = reference, samplesShape = neutralMeshes.toIndexedSeq, samplesColor = neutralMeshes.toIndexedSeq, samplesExpression = neutralWithExpressions.toIndexedSeq, shapeNoiseVariance = 0, colorNoiseVariance = 0, expressionNoiseVariance = 0) println("... initial model is built - (not handling missing color) ...") println(s"... ... shape rank: ${momo.shape.rank}") println(s"... ... color rank: ${momo.color.rank}") println(s"... ... exp rank: ${momo.expression.rank}") val colors: Seq[DiscreteField[_3D, RGBA]] = data.collect({ case Success(e) => e._1 }) val colorModel = buildColorModel(reference, colors.toIndexedSeq, colors.size - 1) println("... color model is built ...") val bu3dModel = MoMo(momo.referenceMesh, momo.shape, colorModel, momo.expression, momo.landmarks) val newModelPath = dataProvider.repositoryRoot / "data" / "modelbuilding" / "model" / s"bu3d-${maskType}.h5" newModelPath.jfile.getParentFile.mkdirs() MoMoIO.write(bu3dModel, newModelPath.jfile) println("... model building finished!") } /** Align vcm to reference. */ def align(reference: TriangleMesh3D, vcm: VertexColorMesh3D): VertexColorMesh3D = { val t: RigidTransformation[_3D] = LandmarkRegistration.rigid3DLandmarkRegistration( vcm.shape.pointSet.points.zip(reference.pointSet.points).toSeq, Point(0, 0, 0) ) val transform = new Transform3D { override def apply(x: Point[_3D]): Point[_3D] = t(x) override def apply(v: Vector[_3D]): Vector[_3D] = t.rotation(v.toPoint).toVector } vcm.transform(transform) } /** Align mesh to reference. */ def alignIt(vcm: VertexColorMesh3D): VertexColorMesh3D = align(reference, vcm) /** Execute function f for neutral and expression in ne. */ def doForBothInNWE(ne: NeutralWithExpression, f: VertexColorMesh3D => VertexColorMesh3D): NeutralWithExpression = { ne match { case NeutralWithExpression(n, e) => NeutralWithExpression(f(n), f(e)) } } /** Mask mesh with precalculated mask. */ def maskMesh(mesh: VertexColorMesh3D): VertexColorMesh3D = { if (mask.isDefined) { val reducer = mesh.shape.operations.maskPoints(mask.get) VertexColorMesh3D( reducer.transformedMesh, SurfacePointProperty.sampleSurfaceProperty(reducer.applyToSurfaceProperty(mesh.color), _.head) ) } else { println("Warning: maskMesh is called but no mask is defined!") println("\t Hence the mesh is left unaltered.") mesh } } /** * Loads the neutral face and the expressions. * * @param reference * @param otherExpressions * @param id * @return */ def prepareData(reference: TriangleMesh3D, otherExpressions: Seq[ExpressionType], id: dataProvider.Person) = { for { unalignedRegistrationWithColor <- dataProvider.model.loadColoredMesh(id, dataProvider.Neutral) } yield { val unaligned = VertexColorMesh3D( unalignedRegistrationWithColor.shape, unalignedRegistrationWithColor.color.asInstanceOf[SurfacePointProperty[RGBA]] ) val masked = if ( mask.isDefined) maskMesh(unaligned) else unaligned val neutral = alignIt(masked) val neutralWithExpressionList = otherExpressions.par.map { exp => println(s"... ... ${exp}") val unalignedExpression = dataProvider.model.loadColoredMesh(id, exp).get val unaligned = VertexColorMesh3D( unalignedExpression.shape, unalignedExpression.color.asInstanceOf[SurfacePointProperty[RGBA]] ) val masked = if (mask.isDefined) maskMesh(unaligned) else unaligned val alignedExpression = alignIt(masked) NeutralWithExpression(neutral, alignedExpression) }.toIndexedSeq val neutralColor = DiscreteField[_3D, RGBA](neutral.shape.pointSet, neutral.color.pointData) ( neutralColor, neutral, neutralWithExpressionList ) } } } /** * Calculate the rigid 3d transform that aligns the mesh to the target. */ def calculateShapeAligningTransform(mesh: TriangleMesh3D, target: TriangleMesh3D): Transform3D = { val t: RigidTransformation[_3D] = LandmarkRegistration.rigid3DLandmarkRegistration( mesh.pointSet.points.zip(target.pointSet.points).toSeq, Point(0,0,0) ) val transform = new Transform3D{ override def apply(x: Point[_3D]): Point[_3D] = t(x) override def apply(v: Vector[_3D]): Vector[_3D] = t.rotation(v.toPoint).toVector } transform } /** * Builds a color model. This model building accounts for missing color values. * * @param referenceMesh Reference mesh. * @param colorFields Colorfields to build the model from. * @param numberOfComponents Number of desired components. * @return Color model. */ def buildColorModel( referenceMesh: TriangleMesh3D, colorFields: IndexedSeq[DiscreteField[_3D, RGBA]], numberOfComponents: Int ): PancakeDLRGP[_3D, RGB] = { val domain = referenceMesh.pointSet val meanRGBA = DiscreteField[_3D, RGBA](domain, saveMean(colorFields.map(_.data))) val meanFreeColors = saveMeanFreeColors(colorFields.map(_.data), meanRGBA.data) val meanFreeColorFields = meanFreeColors.map{ a => DiscreteField(domain, a) } val meanRGB = DiscreteField[_3D, RGB](domain, meanRGBA.data.map(_.toRGB)) val kernel: MatrixValuedPDKernel[_3D] = ReducedEntryKernel(meanFreeColorFields) val gp: GaussianProcess[_3D, RGB] = GaussianProcess(meanRGB.interpolateNearestNeighbor(), kernel) val lrgp = LowRankGaussianProcess.approximateGP[_3D,RGB](gp,UniformMeshSampler3D(referenceMesh,500),numberOfComponents) val grf = lrgp.discretize(domain) PancakeDLRGP(grf) } /** * Calculates the mean based on available samples only. * If no value is available, i.e. the alpha channel is zero for all samples at a given vertex, BlackTransparent is set as mean color. */ def saveMean(colorVectors: Seq[IndexedSeq[RGBA]]): IndexedSeq[RGBA] = { val numberOfColorValues = colorVectors.size val numberOfSamples = colorVectors.head.size val accumulatedColor = Array.fill(numberOfSamples)(RGBA.BlackTransparent) val numberOfUsedColors = Array.fill(numberOfSamples)(0) for ( i <- 0 until numberOfColorValues; j <- 0 until numberOfSamples ) { val color = colorVectors(i)(j) if (color.a == 1.0) { accumulatedColor(j) = accumulatedColor(j) + color numberOfUsedColors(j) += 1 } } val mean = accumulatedColor.zip(numberOfUsedColors).map { case (sumOfColors, counter) => if (counter == 0) RGBA.BlackTransparent else sumOfColors / counter } mean } /** * Substracts the mean color vector from all color samples. Else BlackTransparent. */ def saveMeanFreeColors(colorVectors: Seq[IndexedSeq[RGBA]], meanColorVector: IndexedSeq[RGBA]): Seq[IndexedSeq[RGBA]] = { colorVectors.map { colorVector => colorVector.zip(meanColorVector).map { case (color, meanColor) => if (color.a == 1.0) { val d = RGBA(color.r - meanColor.r, color.g - meanColor.g, color.b - meanColor.b, color.a) d } else { RGBA.BlackTransparent } } } } /** * Covariance kernel that has a backupkernel used when some of the data is missing. * * @param colorFields Input data with possibly some data missing. */ case class MissingEntryKernel( colorFields: Seq[DiscreteField[_3D, RGBA]], backupKernel: MatrixValuedPDKernel[_3D] = DiagonalKernel(GaussianKernel[_3D](10), 3) * 0.0001 ) extends MatrixValuedPDKernel[_3D] { val fs = colorFields.map(f => f.interpolateNearestNeighbor()) override protected def k(x: Point[_3D], y: Point[_3D]): DenseMatrix[Double] = { val correlation = fs.foldLeft(DenseMatrix.zeros[Double](outputDim, outputDim)) { (sum, field) => val xc = field(x) val yc = field(y) val addend = if (xc.a > 0.5 && yc.a > 0.5) { xc.toRGB.toVector.outer(yc.toRGB.toVector).toBreezeMatrix } else { backupKernel(x, y) } sum + addend * (1.0 / fs.size) } correlation } override def domain: Domain[_3D] = RealSpace[_3D] override def outputDim: Int = 3 } /** * Kernel estimating the covariance only on the available data. */ case class ReducedEntryKernel( colorFields: Seq[DiscreteField[_3D, RGBA]], backupKernel: MatrixValuedPDKernel[_3D] = DiagonalKernel(GaussianKernel[_3D](5), 3) * 0.1 ) extends MatrixValuedPDKernel[_3D] { val fs = colorFields.map(f => f.interpolateNearestNeighbor()) private val originalDomain = colorFields.head.domain val countEntries = DiscreteField(originalDomain,colorFields.foldLeft( IndexedSeq.fill[Int](originalDomain.numberOfPoints)(0) ) { case (sum, field) => field.data.map(c => if(c.a==1.0) 1 else 0).zip(sum).map(p => p._1+p._2) }).interpolateNearestNeighbor() override protected def k(x: Point[_3D], y: Point[_3D]): DenseMatrix[Double] = { var count = 0 val correlation = fs.foldLeft(DenseMatrix.zeros[Double](outputDim, outputDim)) { (sum, field) => val xc = field(x) val yc = field(y) if (xc.a > 0.5 && yc.a > 0.5) { val addend = xc.toRGB.toVector.outer(yc.toRGB.toVector).toBreezeMatrix count += 1 sum + addend } else { sum } } if (count>0) { correlation*(1.0/count) } else { backupKernel(x,y) } } override def domain: Domain[_3D] = RealSpace[_3D] override def outputDim: Int = 3 } } ================================================ FILE: src/main/scala/preprocessing/ConvertBu3DRawData.scala ================================================ /* * Copyright University of Basel, Graphics and Vision Research Group * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ package preprocessing import java.awt.image.BufferedImage import java.io.{File, FileInputStream, InputStream} import javax.imageio.ImageIO import ch.unibas.cs.gravis.facepipeline._ import scalismo.faces.landmarks.TLMSLandmark3D import scalismo.faces.mesh.{ColorNormalMesh3D, TextureMappedProperty, VertexPropertyPerTriangle} import scalismo.faces.render.Transform3D import scalismo.faces.utils.ResourceManagement import scalismo.faces.color.RGBA import scalismo.faces.image.PixelImage import scalismo.common.PointId import scalismo.geometry._ import scalismo.mesh._ import scalismo.registration.{LandmarkRegistration, RigidTransformation, TranslationTransform} import scala.io.Source import scala.reflect.io._ import scala.collection.mutable.ListBuffer import scala.util.Try object ConvertBu3DRawData { def main(args: Array[String]) { import ch.unibas.cs.gravis.facepipeline.BU3DDataProvider scalismo.initialize() ConvertBu3DRawData(BU3DDataProvider).run() } } case class ConvertBu3DRawData(dataProvider : DataProvider) extends PipelineStep { override def run() { val inputDirectory = dataProvider.repositoryRoot / "data" / "bu3dfe" / "original" val ids = getIds(inputDirectory) println(s"found ${ids.size} ids....") ids.take(4).foreach { id => // todo: remove take(4) to build on full database preprocessMaximalExpressions(id, inputDirectory) } } def getIds(directory: Path): Seq[dataProvider.Person] = { require( directory.isDirectory, "Expected path to the parent DIRECTORY containing the 100 folders of the BU3D, one for each person." ) val subDirectories = directory.toDirectory.list val subDirectoryNames = subDirectories.map(_.name).toIndexedSeq val sortedList = subDirectoryNames.sortWith((l, r) => l.compareTo(r) < 0) sortedList.map { subdir => val filenames = (directory / subdir).toDirectory.list.map(_.name).toIndexedSeq val filename = filenames.filter(file => file.contains("NE00") && file.contains("RAW") && file.contains(".wrl")).head dataProvider.personFromFilename(filename) } } def getFilestem(id: dataProvider.Person, expression: ExpressionType, mask: MaskType): String = { val base = id + expression.toString s"${id.id}${expression}${id.raceTag}${mask}" } def preprocessMaximalExpressions(id: dataProvider.Person, inputDirectory: Path): Unit = { import BU3DDataProvider.{RAW,F3D,Original,Aligned} val expressions = dataProvider.expressions val outputPath = dataProvider.repositoryRoot / "data" / "incoming" expressions.foreach { expression => Seq(RAW,F3D).map { mask => val meshStem = getFilestem(id, expression, mask) val textureStem = getFilestem(id, expression, F3D) val meshPath = outputPath / "mesh" / s"${id.id}${expression}${id.raceTag}${mask}.ply" if (! meshPath.jfile.exists() ) { println(meshStem + " converting mesh") val texture = loadTexture(id, textureStem, inputDirectory) val mesh = loadBu3DWRLHacky(id, meshStem, inputDirectory, texture) dataProvider.incoming.saveMesh(id, expression, mask, Original, mesh).get } else println(meshStem + " already converted mesh") // convert all available landmarks val landmarkPath = outputPath / "landmarks" / s"${id.id}${expression}${id.raceTag}${mask}.tlms" if (! landmarkPath.jfile.exists() ) { println(meshStem + " converting landmarks") if (meshStem.endsWith("F3D")) { val landmarks = readBU3DFEbnd(inputDirectory / id.id / s"${meshStem}.bnd").get dataProvider.incoming.saveLandmarks(id, expression, mask, Original, landmarks.map { lm => Landmark[_3D](id = lm.id, point = lm.point) }).get } if (meshStem.endsWith("RAW")) { val landmarks = readBU3DFEpse(inputDirectory / id.id / s"${meshStem}.pse").get dataProvider.incoming.saveLandmarks(id, expression, mask, Original, landmarks.map { lm => Landmark[_3D](id = lm.id, point = lm.point) }).get } } else { println(meshStem + " already converted landmarks") } } } // do alignment after converting all data as we need for each id raw+f3d available expressions.foreach { expression => // align f3d data to original raw data val meshStem = getFilestem(id, expression, F3D) val landmarkPath = outputPath / "landmarks" / s"${id.id}${expression}${id.raceTag}${F3D}_aligned.tlms" val meshPath = outputPath / "mesh" / s"${id.id}${expression}${id.raceTag}${F3D}_aligned.ply" if ( (!landmarkPath.jfile.exists()) && (!meshPath.jfile.exists()) ) { println(meshStem + " calculating alignment") val rawMesh = dataProvider.incoming.loadColoredMesh(id, expression, RAW).get val rawLandmarks = dataProvider.incoming.loadLandmarks(id, expression, RAW, Original).get val f3dMesh = dataProvider.incoming.loadColoredMesh(id, expression, F3D).get val f3dLandmarks = dataProvider.incoming.loadLandmarks(id, expression, F3D, Original).get val aligned = align(f3dMesh, f3dLandmarks, rawMesh, rawLandmarks) aligned match { case Some((meshAligned, landmarksAligned)) => dataProvider.incoming.saveLandmarks(id, expression, F3D, Aligned, landmarksAligned).get dataProvider.incoming.saveMesh(id, expression, F3D, Aligned, meshAligned).get case None => } } else { println(meshStem + " already aligned") } } } def loadTexture(id: dataProvider.Person, stem: String, inputDirectory: Path): PixelImage[RGBA] = { import scalismo.faces.image.BufferedImageConverter val textureName = inputDirectory / id.id / s"${stem}.bmp" println(textureName.toString()) val img: BufferedImage = ImageIO.read(new java.io.File(textureName.toString())) BufferedImageConverter.toPixelImage(img) } def loadBu3DWRLHacky(id: dataProvider.Person, stem: String, inputDirectory: Path, texture: PixelImage[RGBA]): ColorNormalMesh3D = { val meshName = inputDirectory / id.id / s"$stem.wrl" val lines = Source.fromFile(meshName.toString()).getLines() val coordinates = ListBuffer[Point[_3D]]() val textureCoordinates = ListBuffer[Point[_2D]]() val textureCoordinateIndex = ListBuffer[TriangleCell]() val triangleVertexIndex = ListBuffer[TriangleCell]() parseBu3DWRL(lines, coordinates, triangleVertexIndex, textureCoordinates, textureCoordinateIndex) val vc = coordinates.toIndexedSeq val tc = textureCoordinates.toIndexedSeq val tvi = TriangleList(triangleVertexIndex.toIndexedSeq) val tci = TriangleList(textureCoordinateIndex.toIndexedSeq) val mesh = TriangleMesh3D(vc, tvi) val texCoords = VertexPropertyPerTriangle(tvi, tci.triangles.map(_.toIntVector3D), tc) val tex = TextureMappedProperty(tvi, texCoords, texture) ColorNormalMesh3D(mesh, tex, mesh.cellNormals) } def readBU3DFEbnd(path: Path): Try[IndexedSeq[TLMSLandmark3D]] = { ResourceManagement.usingTry(Try(new FileInputStream(path.toString())))(readBU3DFEbnd) } def readBU3DFEbnd(stream: InputStream): Try[IndexedSeq[TLMSLandmark3D]] = Try { // little bit unsafe, we read each line and expect to have a landmark! var counter = 0 val lines = Source.fromInputStream(stream).getLines() lines.map { line => val fields = line.split("\\s+").map(_.trim) counter += 1 val name = "bu3dfe-lm-" + counter val x = fields(1).toFloat val y = fields(2).toFloat val z = fields(3).toFloat TLMSLandmark3D(name, Point(x, y, z), true) }.toIndexedSeq } def readBU3DFEpse(path: Path): Try[IndexedSeq[TLMSLandmark3D]] = { ResourceManagement.usingTry(Try(new FileInputStream(path.toString())))(readBU3DFEpse) } def readBU3DFEpse(stream: InputStream): Try[IndexedSeq[TLMSLandmark3D]] = Try { // little bit unsafe, we read each line and expect to have an landmark var counter = 0 val lines = Source.fromInputStream(stream).getLines().toIndexedSeq val idOrder = Array(1, 5, 9, 13, 40, 45, 84, 85).toIndexedSeq // filter out pose normal (entry starting with n) val linesFiltered = lines.filterNot(_.substring(0, 1) == "n") if (linesFiltered.length == 8) { for (line <- linesFiltered) yield { val fields = line.split("\\s+").map(_.trim) val name = "bu3dfe-lm-" + idOrder(counter) counter += 1 val x = fields(1).toFloat val y = fields(2).toFloat val z = fields(3).toFloat TLMSLandmark3D(name, Point(x, y, z), true) } } else { // empty list IndexedSeq[TLMSLandmark3D]() } } def align( f3dMeshOrig: ColorNormalMesh3D, f3dLandmarks: Seq[Landmark[_3D]], rawMeshOrig: ColorNormalMesh3D, rawLandmarks: Seq[Landmark[_3D]] ): Option[(ColorNormalMesh3D, Seq[Landmark[_3D]])] = { if (f3dLandmarks.isEmpty || rawLandmarks.isEmpty) { println(s"Not enough landmarks: f3dLandmarks.size=${f3dLandmarks.size} / rawLandmarks.size=${rawLandmarks.size}") return None } // landmark transformation val transLM = LandmarkRegistration.rigid3DLandmarkRegistration(f3dLandmarks, rawLandmarks, Point(0,0,0)) val transLMNormal: RigidTransformation[_3D] = RigidTransformation(transLM.rotation, TranslationTransform(Vector(0, 0, 0))) val transLMFaces: Transform3D = new Transform3D { override def apply(x: Point[_3D]): Point[_3D] = transLM.f(x) override def apply(v: Vector[_3D]): Vector[_3D] = transLMNormal.f(v.toPoint).toVector } var f3dLandmarksT1 = transformLandmarks(f3dLandmarks, transLM) var f3dPointsT1 = f3dMeshOrig.transform(transLMFaces) val n0 = for (p <- f3dPointsT1.shape.pointSet.points) yield { rawMeshOrig.shape.pointSet.findClosestPoint(p).point } var avgError = f3dPointsT1.shape.pointSet.points.zip(n0).map(p => (p._1 - p._2).norm2).sum / f3dMeshOrig.shape.pointSet.points.toIndexedSeq.length var iteration = 0 var r = scala.util.Random while (avgError > 0.2 && iteration < 500) { val f3DPointsSampled = for (i <- 1 to 1000) yield { f3dPointsT1.shape.pointSet.pointsWithId.toIndexedSeq(r.nextInt(f3dPointsT1.shape.pointSet.points.length - 1)) } val nn = for (f3dPoint <- f3DPointsSampled) yield { val rawNeighbor = rawMeshOrig.shape.pointSet.findClosestPoint(f3dPoint._1) if ((f3dPoint._1 - rawNeighbor.point).norm < 5) { (f3dPoint._1, rawNeighbor.point) } else { (None, None) } } val nnFiltered = nn.filterNot(p => p._1 == None || p._2 == None).map(p => (p._1.asInstanceOf[Point3D], p._2.asInstanceOf[Point3D])) if (nnFiltered.length > 0) { val transT2 = LandmarkRegistration.rigid3DLandmarkRegistration(nnFiltered, Point(0,0,0)) val transT2Normal: RigidTransformation[_3D] = RigidTransformation(transT2.rotation, TranslationTransform(Vector(0, 0, 0))) val transT2Faces: Transform3D = new Transform3D { override def apply(x: Point[_3D]): Point[_3D] = transT2.f(x) override def apply(v: Vector[_3D]): Vector[_3D] = transT2Normal.f(v.toPoint).toVector } val f3dLandmarksT2 = transformLandmarks(f3dLandmarksT1, transT2) val f3dPointsT2 = f3dPointsT1.transform(transT2Faces) val sampledPointsTrans = transformPoints(nnFiltered.map(p => p._1), transT2) avgError = sampledPointsTrans.zip(nnFiltered).map(p => (p._1 - p._2._2).norm).sum / sampledPointsTrans.length f3dPointsT1 = f3dPointsT2 f3dLandmarksT1 = f3dLandmarksT2 iteration = iteration + 1 } else { println("icp alignment did not find corresponding points!") avgError = -1 } } println("... alignment error after iteration " + iteration + ": " + avgError) val outTLMSLandmarks = for (lm <- f3dLandmarksT1.zip(f3dLandmarks)) yield { TLMSLandmark3D(lm._2.id, lm._1.point, true) } Some((f3dPointsT1, f3dLandmarksT1)) } def transformLandmarks(landmarks: Seq[Landmark[_3D]], trans: RigidTransformation[_3D]): Seq[Landmark[_3D]] = { for (lm <- landmarks) yield { Landmark[_3D](lm.id, trans(lm.point), lm.description, lm.uncertainty) } } def transformPoints(points: Seq[Point[_3D]], trans: RigidTransformation[_3D]): Seq[Point[_3D]] = { for (p <- points) yield { trans(p) } } def parseBu3DWRL( lines: Iterator[String], coordinates: ListBuffer[Point[_3D]], tvi: ListBuffer[TriangleCell], textureCoordinates: ListBuffer[Point[_2D]], tci: ListBuffer[TriangleCell] ): Unit = { val BOILERPLATE = "boilerplate" val COORDINATES = "coord Coordinate" val COORDINATES2 = "coord Coordinate {" val TEXTURECOORDS = "texCoord TextureCoordinate" val TEXTURECOORDS2 = "texCoord TextureCoordinate {" val TEXCOORDINDEX = "texCoordIndex [" val VERTEXINDEX = "coordIndex [" var mode = "boilerplate" for (line <- lines) { val trimmed = line.trim mode match { case BOILERPLATE => { trimmed match { case COORDINATES => mode = COORDINATES case COORDINATES2 => mode = COORDINATES case TEXTURECOORDS => mode = TEXTURECOORDS case TEXTURECOORDS2 => mode = TEXTURECOORDS case TEXCOORDINDEX => mode = TEXCOORDINDEX case VERTEXINDEX => mode = VERTEXINDEX case _ => } } case COORDINATES => { trimmed match { case "{" => case "point" => case "point [" => case "[" => case "" => case "]" => mode = BOILERPLATE case data => { data.split(',').map(_.trim).filter(s => s.nonEmpty).foreach { tripplet => val point = Point[_3D](tripplet.split(" ").map { w => try { w.toDouble } catch { case e: Throwable => { println(w) throw (e) } } }) coordinates += point } } } } case TEXTURECOORDS => { trimmed match { case "{" => case "point" => case "point [" => case "[" => case "" => case "]" => mode = BOILERPLATE case data => { data.split(",").map(_.trim).filter(_.nonEmpty).flatMap(_.split(" ")).map(_.trim).filter(_.nonEmpty).grouped(2).foreach { pair => val point = Point[_2D](pair.map { w => try { w.toDouble } catch { case e: Throwable => { println(w) throw (e) } } }) textureCoordinates += point } } } } case VERTEXINDEX => { trimmed match { case "[" => case "" => case "]" => mode = BOILERPLATE case data => { val stringToParse = if (data.last == ',') data.init else data val indices = stringToParse.split(" ").map(_.trim).filter(_.nonEmpty).map(_.toInt).toIndexedSeq require(indices.last == -1) val alwaysFirst = indices.head val firstAndlastRemoved = indices.init.tail val seconds = firstAndlastRemoved.init val thirds = firstAndlastRemoved.tail val grouped = seconds.zip(thirds) grouped.foreach { pair => tvi += TriangleCell( PointId(alwaysFirst), PointId(pair._1), PointId(pair._2) ) } } } } case TEXCOORDINDEX => { trimmed match { case "[" => case "" => case "]" => mode = BOILERPLATE case data => { val stringToParse = if (data.last == ',') data.init else data val indices = stringToParse.split(" ").map(_.trim).filter(_.nonEmpty).map(_.toInt).toIndexedSeq require(indices.last == -1) val alwaysFirst = indices.head val firstAndlastRemoved = indices.init.tail val seconds = firstAndlastRemoved.init val thirds = firstAndlastRemoved.tail val grouped = seconds.zip(thirds) grouped.foreach { pair => tci += TriangleCell( PointId(alwaysFirst), PointId(pair._1), PointId(pair._2) ) } } } } } } } } ================================================ FILE: src/main/scala/preprocessing/PrepareReferenceLandmarks.scala ================================================ /* * Copyright University of Basel, Graphics and Vision Research Group * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ package preprocessing import breeze.linalg.{DenseMatrix, DenseVector} import ch.unibas.cs.gravis.facepipeline.BU3DDataProvider import ch.unibas.cs.gravis.facepipeline.BU3DDataProvider.Expressions import ch.unibas.cs.gravis.facepipeline.{DataProvider, PipelineStep} import scalismo.faces.io.TLMSLandmarksIO import scalismo.statisticalmodel.MultivariateNormalDistribution object PrepareReferenceLandmarks { def main(args: Array[String]): Unit = { scalismo.initialize() PrepareReferenceLandmarks(BU3DDataProvider).run() } } case class PrepareReferenceLandmarks(dataProvider : DataProvider) extends PipelineStep { override def run(): Unit = { scalismo.initialize() val rawRefLmsFile = (dataProvider.repositoryRoot / "data" / "incoming" / "reference" / "landmarks" / "mean2012_l7_bfm_nomouth.tlms").jfile val referenceLandmarksTLMS = TLMSLandmarksIO.read3D(rawRefLmsFile).get val referenceLandmarks = for (lmTlms <- referenceLandmarksTLMS if lmTlms.visible) yield { val lm = lmTlms.toLandmark val noiseVariance = lm.id.trim match { case lmid if lmid.contains("eyebrow") => 3.0 case lmid if lmid.contains("eye.bottom") => 3.0 case lmid if lmid.contains("eye.top") => 3.0 case _ => 1.0 } lm.copy(uncertainty = Some(MultivariateNormalDistribution(DenseVector.zeros[Double](3), DenseMatrix.eye[Double](3) * noiseVariance))) } // Transfer the reference landmarks to all the expressions and save them. for (expression <- Expressions.expressionModelTypes()) { val neutralRef = dataProvider.incoming.reference.loadMesh(dataProvider.Neutral).get val expressionRef = dataProvider.registration.loadPriorModel(expression).get.referenceMesh val expressionLms = for (lm <- referenceLandmarks) yield { val id = neutralRef.pointSet.findClosestPoint(lm.point).id lm.copy(point = expressionRef.pointSet.point(id)) } dataProvider.incoming.reference.saveLandmarks(expression, expressionLms) } } } ================================================ FILE: src/main/scala/registration/Registration.scala ================================================ /* * Copyright University of Basel, Graphics and Vision Research Group * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ package registration import breeze.linalg.DenseVector import ch.unibas.cs.gravis.facepipeline._ import com.typesafe.scalalogging.LazyLogging import ch.unibas.cs.gravis.facepipeline.BU3DDataProvider._ import _root_.registration.utils.VisualLogger import _root_.registration.modelbuilding.FaceMask import _root_.registration.metrics.HuberDistanceMetric import scalismo.common.{PointId, UnstructuredPointsDomain} import scalismo.geometry.{Landmark, Point, _3D} import scalismo.mesh.{MeshBoundaryPredicates, TriangleMesh, TriangleMesh3DOperations} import scalismo.numerics.{LBFGSOptimizer, Sampler, UniformMeshSampler3D} import scalismo.registration._ import scalismo.statisticalmodel.{DiscreteLowRankGaussianProcess, StatisticalMeshModel} import scalismo.utils.Random case class Registration(dataProvider: DataProvider) extends PipelineStep with LazyLogging { type CoefficientVector = DenseVector[Double] case class LandmarkPair(referenceLandmark: Landmark[_3D], targetLandmark: Landmark[_3D]) case class LevelConfig(regularizationWeight : Double, outlierThreshold : Option[Double], numBasisFunctions : Int) case class OutlierAwarePointSampler(referenceMesh: TriangleMesh[_3D], sampledNumberOfPoints: Int, isValidTargetPoint: Point[_3D] => Boolean)(implicit rand: Random) extends Sampler[_3D] with LazyLogging { private val points = UniformMeshSampler3D(referenceMesh, sampledNumberOfPoints).sample()(rand).map(_._1) private val validPointsOnly = points.filter(isValidTargetPoint) override val numberOfPoints: Int = validPointsOnly.size logger.info(s"sampling $numberOfPoints points") override def volumeOfSampleRegion: Double = referenceMesh.area override def sample()(implicit rand: Random): IndexedSeq[(Point[_3D], Double)] = { validPointsOnly.map(p => (p, 1.0 / referenceMesh.area)) } } def registration(gpModel: StatisticalMeshModel, targetMesh: TriangleMesh[_3D], faceMask : FaceMask, landmarkPairs: Seq[LandmarkPair]): TriangleMesh[_3D] = { val referenceMesh = gpModel.referenceMesh VisualLogger.showTargetMesh(targetMesh) val landmarkConstraints = for (landmarkPair <- landmarkPairs.toIndexedSeq) yield { val referencePointId = referenceMesh.pointSet.findClosestPoint(landmarkPair.referenceLandmark.point).id val targetPoint = landmarkPair.targetLandmark.point (referencePointId, targetPoint, landmarkPair.referenceLandmark.uncertainty.get) } val posteriorModel = gpModel.posterior(landmarkConstraints) VisualLogger.ui.map(_.show(posteriorModel,"M")) var initialCoefficients = DenseVector.zeros[Double](posteriorModel.rank) val levelConfigs = Seq(LevelConfig(1.0, None, gpModel.rank), LevelConfig(1E-1, None, gpModel.rank), LevelConfig(1E-3, None, gpModel.rank), LevelConfig(1E-4, Some(4.0), gpModel.rank), LevelConfig(1E-5, Some(2.0), gpModel.rank), LevelConfig(1E-6, Some(1.0), gpModel.rank) ) val finalCoefficients = levelConfigs.foldLeft[DenseVector[Double]](initialCoefficients){ case(currentCoefficients, levelConfig) => { registrationForLevel(posteriorModel, targetMesh, faceMask, levelConfig, numberOfIterations = 20, currentCoefficients) } } posteriorModel.instance(finalCoefficients) } def registrationForLevel(gpModel: StatisticalMeshModel, targetMesh : TriangleMesh[_3D], faceMask: FaceMask, levelConfig : LevelConfig, numberOfIterations: Int, initialCoefficients: CoefficientVector): CoefficientVector = { val LevelConfig(regularizationWeight, outlierThreshold, numBasisFunctions) = levelConfig val reducedGPModel = reduceModel(gpModel, numBasisFunctions) val reducedInitialCoefficients = initialCoefficients(0 until numBasisFunctions) val referenceMesh = reducedGPModel.referenceMesh val currentFit = reducedGPModel.instance(reducedInitialCoefficients) VisualLogger.showStatisticalShapeModel(reducedGPModel) VisualLogger.updateModelView(reducedInitialCoefficients) // here we need to compute a new posterior based on the line landmarks def isValidTargetPoint(currentFit: TriangleMesh[_3D], targetMeshOps: TriangleMesh3DOperations, targetMeshBoundary: UnstructuredPointsDomain[_3D]) (p: Point[_3D]): Boolean = { val ptId = referenceMesh.pointSet.findClosestPoint(p).id val closestPt = targetMeshOps.closestPointOnSurface(currentFit.pointSet.point(ptId)) val closestPtId = targetMesh.pointSet.findClosestPoint(closestPt.point).id def isOnValidBoundary(ptId : PointId, closestPtId : PointId) : Boolean = { if(faceMask.isLipPoint(ptId)) { true } else { (closestPt.point - targetMeshBoundary.findClosestPoint(closestPt.point).point).norm > 8.0 // Points that are close to a border } } def getOutlierTreshold(ptId : PointId) : Double = { if(faceMask.isLipPoint(ptId)) { Double.MaxValue } else { outlierThreshold.getOrElse(Double.MaxValue) } } Math.sqrt(closestPt.distanceSquared) < getOutlierTreshold(ptId) && isOnValidBoundary(ptId,closestPtId) && !faceMask.isEarRegion(ptId) && !faceMask.isNoseRegion(ptId) } val targetMeshBoundaryPred = MeshBoundaryPredicates(targetMesh) val targetMeshBoundary = UnstructuredPointsDomain(targetMesh.pointSet.pointIds .filter(targetMeshBoundaryPred.pointIsOnBoundary) .map(targetMesh.pointSet.point).toIndexedSeq ) val optimizationPointSampler = OutlierAwarePointSampler(referenceMesh, sampledNumberOfPoints = referenceMesh.pointSet.numberOfPoints, isValidTargetPoint(currentFit, targetMesh.operations, targetMeshBoundary)) val config = RegistrationConfiguration[_3D, GaussianProcessTransformationSpace[_3D]]( optimizer = LBFGSOptimizer(numIterations = numberOfIterations), metric = HuberDistanceMetric[_3D](optimizationPointSampler), transformationSpace = GaussianProcessTransformationSpace(reducedGPModel.gp.interpolateNearestNeighbor), regularizer = L2Regularizer, regularizationWeight = regularizationWeight) // Scalismo implements registration always as image to image registration. // Therefore we compute distance images from the meshes val fixedImage = referenceMesh.operations.toDistanceImage val movingImage = targetMesh.operations.toDistanceImage val registrationIterator = scalismo.registration.Registration.iterations(config)(fixedImage, movingImage, reducedInitialCoefficients) val iteratorWithLogging = for ((regState, itNum) <- registrationIterator.zipWithIndex) yield { logger.debug(s"Iteration $itNum: value = ${regState.optimizerState.value}") VisualLogger.updateModelView(regState.optimizerState.parameters) regState } val lastRegistrationState = iteratorWithLogging.toSeq.last val fullFinalParameters = DenseVector.zeros[Double](initialCoefficients.length) fullFinalParameters(0 until numBasisFunctions) := lastRegistrationState.optimizerState.parameters fullFinalParameters } private def reduceModel(model : StatisticalMeshModel, numBasisFunctions : Int) : StatisticalMeshModel = { val reducedGp = DiscreteLowRankGaussianProcess(model.gp.mean, model.gp.klBasis.take(numBasisFunctions)) model.copy(gp = reducedGp) } override def run(): Unit = { // transforms the mesh using the best similarity transform between the reference and target landmarks. for (expression <- Seq(Neutral,Sadness,Joy,Disgust,Anger,Fear,Surprise).reverse) { val referenceLandmarks = dataProvider.incoming.reference.loadLandmarks(expression = if(expression == Neutral) Neutral else CoreExpression).get val model = dataProvider.registration.loadPriorModel(expression = if(expression == Neutral) Neutral else CoreExpression).get val faceMask = dataProvider.incoming.reference.loadFaceMask().get logger.info("Successfully loaded reference and model") for (id <- scala.util.Random.shuffle(dataProvider.incoming.ids(expression)) if dataProvider.registration.loadMesh(id,expression).isFailure && dataProvider.incoming.loadLandmarks(id,expression).isSuccess) { logger.info("Performing registration for id " + id) val targetMesh = dataProvider.incoming.loadMesh(id,expression).get val targetLandmarks = dataProvider.incoming.loadLandmarks(id,expression).get val correspondingLandmarks = correspondingLandmarkPairs(referenceLandmarks, targetLandmarks) val correspondingLandmarkPoints = correspondingLandmarks.map(lmPair => (lmPair.targetLandmark.point, lmPair.referenceLandmark.point)) val alignmentTransform = LandmarkRegistration.similarity3DLandmarkRegistration(correspondingLandmarkPoints, center = Point(0.0, 0.0, 0.0)) val alignedTargetMesh = targetMesh.transform(alignmentTransform) val alignedLandmarkPairs = correspondingLandmarks.map(lmPair => LandmarkPair(lmPair.referenceLandmark, lmPair.targetLandmark.transform(alignmentTransform)) ) VisualLogger.ui.map(_.show(alignedLandmarkPairs.map(_.targetLandmark),"Test")) VisualLogger.ui.map(_.show(alignedLandmarkPairs.map(_.referenceLandmark),"Test")) val registeredMesh = registration(model, alignedTargetMesh, faceMask, alignedLandmarkPairs) // we realign the registered mesh with the target. val registeredMeshOrigSpace = registeredMesh.transform(alignmentTransform.inverse) dataProvider.registration.saveMesh(id,expression, registeredMeshOrigSpace) } } } private def correspondingLandmarkPairs(referenceLandmarks: Seq[Landmark[_3D]], targetLandmarks: Seq[Landmark[_3D]]): Seq[LandmarkPair] = { referenceLandmarks .map(refLm => (refLm, targetLandmarks.find(targetLm => targetLm.id == refLm.id))) .filter(lmTuple => lmTuple._2.nonEmpty) .map(lmTuple => LandmarkPair(lmTuple._1, lmTuple._2.get)) } } object Registration { def main(args: Array[String]): Unit = { scalismo.initialize() Registration(BU3DDataProvider).run() } } ================================================ FILE: src/main/scala/registration/experiments/Bu3DFELandmarkEvaluation.scala ================================================ /* * Copyright University of Basel, Graphics and Vision Research Group * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ package registration.experiments import ch.unibas.cs.gravis.facepipeline.BU3DDataProvider.Neutral import ch.unibas.cs.gravis.facepipeline.BU3DDataProvider._ import scalismo.geometry.{Landmark,_3D} import scalismo.io.LandmarkIO import java.io.File import breeze.linalg.DenseVector import breeze.numerics.sqrt import ch.unibas.cs.gravis.facepipeline.BU3DDataProvider import scalismo.common.PointId case class FeaturePoints(f : Feature, points : Seq[PointId], lms: Seq[Landmark[_3D]]) case class Feature(lmids: Seq[Int],name: String) { override def toString: String = name } object Bu3DFELandmarkEvaluation { def main(args: Array[String]): Unit = { scalismo.initialize() val dataProvider = BU3DDataProvider val eyeLeft = Feature(Seq(2,3,4,6,7,8),"eye.left") val eyeRight = Feature(Seq(10,11,12,14,15,16),"eye.right") val eyebrowLeft = Feature((17 to 26),"eyebrow.left") val eyebrowRight = Feature((27 to 36),"eyebrow.right") val nose = Feature((37 to 47),"nose") val mouth = Feature((49 to 60),"mouth") val faceRight = Feature((69 to 74),"face.right") val faceLeft = Feature((78 to 83),"face.left") val chin = Feature((75 to 77),"chin") val features = Seq(eyeLeft,eyeRight,eyebrowLeft,eyebrowRight,nose,mouth,faceRight,faceLeft,chin) val referenceMesh = dataProvider.incoming.reference.loadMesh(Neutral).get val bu3DreferenceLandmarks = { LandmarkIO.readLandmarksCsv[_3D](new File(dataProvider.repositoryRoot.jfile,"/data/incoming/reference/landmarks/mean2012-bu3dfe-eval-landmarks.csv")).get } val featuresWithLandmarks = features.map{ f => val ids = f.lmids val lms = ids.map(id => bu3DreferenceLandmarks.find(lm => lm.id.replace("bu3dfe-lm-","") == s"$id").get) val points = lms.map(lm => referenceMesh.pointSet.findClosestPoint(lm.point).id) FeaturePoints(f,points,lms) } val finalRes = for (expression <- Seq(Neutral,Sadness,Joy,Disgust,Anger,Fear,Surprise).reverse) yield { val evalPerSample = for (id <- dataProvider.incoming.ids(expression) if dataProvider.incoming.loadLandmarks(id,expression, F3D, BU3DDataProvider.Aligned).isSuccess && dataProvider.incoming.loadLandmarks(id, expression).isSuccess) yield { val bu3Dlandmarks = dataProvider.incoming.loadLandmarks(id, expression, F3D, BU3DDataProvider.Aligned).get val registeredMesh = dataProvider.registration.loadMesh(id, expression).get val alldists = for (f <- featuresWithLandmarks) yield { val dists = for (id <- f.lms.zip(f.points)) yield { val registeredPoint = registeredMesh.pointSet.point(id._2) val gtPoint = bu3Dlandmarks.find(lm => lm.id == id._1.id).get.point val dist = sqrt((registeredPoint - gtPoint).norm2) dist } (dists.sum / dists.length) } alldists } evalPerSample } println("Landmark Evaluation Result:") val table = for((f,i) <- features.zipWithIndex) yield { val data = DenseVector(finalRes.map(d => d.map(_(i))).flatten.toArray) val error = breeze.stats.meanAndVariance(data) Seq(f.name,s"${error.mean}",s"${error.stdDev}") } println(Tabulator.format(Seq(Seq("Region","Mean","Std")) ++ table)) } } object Tabulator { def format(table: Seq[Seq[Any]]) = table match { case Seq() => "" case _ => val sizes = for (row <- table) yield (for (cell <- row) yield if (cell == null) 0 else cell.toString.length) val colSizes = for (col <- sizes.transpose) yield col.max val rows = for (row <- table) yield formatRow(row, colSizes) formatRows(rowSeparator(colSizes), rows) } def formatRows(rowSeparator: String, rows: Seq[String]): String = ( rowSeparator :: rows.head :: rowSeparator :: rows.tail.toList ::: rowSeparator :: List()).mkString("\n") def formatRow(row: Seq[Any], colSizes: Seq[Int]) = { val cells = (for ((item, size) <- row.zip(colSizes)) yield if (size == 0) "" else ("%" + size + "s").format(item)) cells.mkString("|", "|", "|") } def rowSeparator(colSizes: Seq[Int]) = colSizes map { "-" * _ } mkString("+", "+", "+") } ================================================ FILE: src/main/scala/registration/metrics/HuberDistanceMetric.scala ================================================ /* * Copyright University of Basel, Graphics and Vision Research Group * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ package registration.metrics import scalismo.common.Domain import scalismo.geometry.{Dim, NDSpace, Point} import scalismo.image.{DifferentiableScalarImage, ScalarImage} import scalismo.numerics.Sampler import scalismo.registration.{ImageMetric, Transformation} case class HuberDistanceMetric[D <: Dim: NDSpace](sampler: Sampler[D]) extends ImageMetric[D] { override val ndSpace = implicitly[NDSpace[D]] def value(fixedImage: ScalarImage[D], movingImage: ScalarImage[D], transform: Transformation[D]) = { val warpedImage = fixedImage.compose(transform) def rhoHuber(v : Float ) : Float = { val k = 1.345 if (v < k) (v * v / 2f) / (1 + v * v) else (k * ( Math.abs(v) - k / 2 )).toFloat } integrator.integrateScalar((warpedImage - movingImage).andThen(rhoHuber _)) / integrator.sampler.volumeOfSampleRegion } def takeDerivativeWRTToTransform(fixedImage: DifferentiableScalarImage[D], movingImage: ScalarImage[D], transform: Transformation[D]) = { def psiHuber(v : Float) : Float = { val k = 1.345 if (v < k) v else (k * Math.signum(v)).toFloat } val movingGradientImage = fixedImage.differentiate val warpedImage = fixedImage.compose(transform) val dDMovingImage = (warpedImage - movingImage).andThen(psiHuber _) * (1.0 / sampler.volumeOfSampleRegion) val fullMetricGradient = (x: Point[D]) => { val domain = Domain.intersection(warpedImage.domain, dDMovingImage.domain) if (domain.isDefinedAt(x)) Some(movingGradientImage(transform(x)).toFloatBreezeVector * dDMovingImage(x)) else None } fullMetricGradient } } ================================================ FILE: src/main/scala/registration/modelbuilding/BuildCoreExpressionPrior.scala ================================================ /* * Copyright University of Basel, Graphics and Vision Research Group * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ package registration.modelbuilding import ch.unibas.cs.gravis.facepipeline.BU3DDataProvider._ import ch.unibas.cs.gravis.facepipeline.{BU3DDataProvider, DataProvider, PipelineStep} import com.typesafe.scalalogging.StrictLogging import scalismo.common._ import scalismo.geometry.{Point, Vector, _3D} import scalismo.statisticalmodel._ case class BuildCoreExpressionPrior(dataProvider: DataProvider) extends PipelineStep with StrictLogging { def run(): Unit = { scalismo.initialize() val dataProvider = BU3DDataProvider logger.info("load reference mesh ...") val referenceMesh = dataProvider.incoming.reference.loadMesh(Neutral).get logger.info("make core model from expressions ...") val references = for(exp <- Seq(Neutral, Sadness, Joy, Disgust, Anger, Fear, Surprise) ) yield { val expression = dataProvider.incoming.reference.loadMesh(exp).get def t(p : Point[_3D]) : Vector[_3D] = { val id = referenceMesh.pointSet.findClosestPoint(p).id val cp = expression.pointSet.point(id) cp - p } Field[_3D,Vector[_3D]](RealSpace[_3D],t) } logger.info("augment neutral with core model ...") val ssm = StatisticalMeshModel.createUsingPCA(referenceMesh,references) val neutralModel = dataProvider.registration.loadPriorModel(Neutral).get val augmentedExpressionModel = StatisticalMeshModel.augmentModel(ssm,neutralModel.gp.interpolateNearestNeighbor) logger.info("save core model ...") dataProvider.registration.savePriorModel(augmentedExpressionModel, dataProvider.CoreExpression) } } object BuildCoreExpressionPrior { def main(args: Array[String]): Unit = { BuildCoreExpressionPrior(BU3DDataProvider).run() } } ================================================ FILE: src/main/scala/registration/modelbuilding/BuildNeutralPrior.scala ================================================ /* * Copyright University of Basel, Graphics and Vision Research Group * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ package registration.modelbuilding import breeze.linalg.DenseMatrix import ch.unibas.cs.gravis.facepipeline.BU3DDataProvider._ import ch.unibas.cs.gravis.facepipeline._ import com.typesafe.scalalogging.StrictLogging import scalismo.common._ import scalismo.geometry.{Vector, _3D} import scalismo.numerics.PivotedCholesky import scalismo.numerics.PivotedCholesky.StoppingCriterion import scalismo.statisticalmodel._ case class BuildNeutralPrior(dataProvider: DataProvider) extends PipelineStep with StrictLogging { def approximatePointSet(points: UnstructuredPointsDomain[_3D], D: Double, gp : GaussianProcess[_3D,Vector[_3D]], sc: StoppingCriterion) : (DiscreteLowRankGaussianProcess[_3D,Vector[_3D]],UnstructuredPointsDomain[_3D]) = { def phiWithDim(i: Int, dim : Int, ptId : Int, phi: DenseMatrix[Double]) = { phi(ptId*3 + dim,i) } def phiVec(i : Int, ptID : PointId,phi : DenseMatrix[Double]) = { Vector(phiWithDim(i,0,ptID.id,phi),phiWithDim(i,1,ptID.id,phi),phiWithDim(i,2,ptID.id,phi)) } val (phi,lambda) = PivotedCholesky.computeApproximateEig(gp.cov,points.points.toIndexedSeq,D,sc) val nPhi = phi.cols val klBasis: DiscreteLowRankGaussianProcess.KLBasis[_3D, Vector[_3D]] = for(i <- 0 until nPhi) yield { val v = DiscreteField[_3D,Vector[_3D]](points,points.pointsWithId.toIndexedSeq.map(f => phiVec(i,f._2,phi))) DiscreteLowRankGaussianProcess.Eigenpair(lambda(i),v) } val mean = DiscreteField[_3D,Vector[_3D]](points,points.points.toIndexedSeq.map(p => gp.mean(p))) val r = DiscreteLowRankGaussianProcess[_3D,Vector[_3D]](mean, klBasis) (r,points) } override def run(): Unit = { scalismo.initialize() logger.info(s"building model for neutral expression") val referenceMesh = dataProvider.incoming.reference.loadMesh(Neutral).get val mask = dataProvider.incoming.reference.loadFaceMask().get val faceKernel = FaceKernel(mask,referenceMesh) val gp = GaussianProcess[_3D, Vector[_3D]](faceKernel) val ldg = approximatePointSet(referenceMesh.pointSet, 1.0, gp, PivotedCholesky.NumberOfEigenfunctions(1000)) val lowRankGaussianProcess = ldg._1.interpolateNearestNeighbor logger.info("computed nystrom approximation") val model = StatisticalMeshModel(referenceMesh, lowRankGaussianProcess) dataProvider.registration.savePriorModel(model, Neutral) logger.info("model building done") } } object BuildNeutralPrior { def main(args: Array[String]): Unit = { BuildNeutralPrior(BU3DDataProvider).run() } } ================================================ FILE: src/main/scala/registration/modelbuilding/FaceKernel.scala ================================================ /* * Copyright University of Basel, Graphics and Vision Research Group * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ package registration.modelbuilding import breeze.linalg.DenseMatrix import registration.modelbuilding.FaceKernel.LevelWithScale import scalismo.common._ import scalismo.geometry.{Point, SquareMatrix, _3D} import scalismo.kernels.{BSplineKernel, DiagonalKernel, MatrixValuedPDKernel} import scalismo.mesh.TriangleMesh case class SpatiallyVaryingMultiscaleKernel(levelsWithScale : Seq[LevelWithScale], mask: FaceMask, referenceMesh: TriangleMesh[_3D]) extends MatrixValuedPDKernel[_3D] { val bSplineKernel = DiagonalKernel(BSplineKernel[_3D](order = 3, scale = 0), 3) val smoothedRegionWeights = levelsWithScale.map(levelWithScale => (levelWithScale.level, mask.computeSmoothedRegions(referenceMesh,levelWithScale.level, 40)) ).toMap def k(x: Point[_3D], y: Point[_3D]): DenseMatrix[Double] = { var sum = SquareMatrix.zeros[_3D].toBreezeMatrix for (LevelWithScale(level, scale) <- levelsWithScale) { val weightX = smoothedRegionWeights(level)(x) val weightY = smoothedRegionWeights(level)(y) sum += bSplineKernel((x.toVector * Math.pow(2, level)).toPoint, (y.toVector * Math.pow(2, level)).toPoint) * scale * weightX * weightY } sum } override def outputDim = 3 override def domain = RealSpace[_3D] } case class FaceKernel(faceMask : FaceMask, referenceMesh: TriangleMesh[_3D]) extends MatrixValuedPDKernel[_3D] { private val faceKernel = { val levelsAndScales = Seq( LevelWithScale(-6,128.0), LevelWithScale(-5, 64.0), LevelWithScale(-4, 32.0), LevelWithScale(-3, 10.0), LevelWithScale(-2, 4.0)) val spatiallyVaryingKernel = SpatiallyVaryingMultiscaleKernel(levelsAndScales, faceMask, referenceMesh) val symmetricKernel = symmetrize(spatiallyVaryingKernel) symmetricKernel * 0.7 + spatiallyVaryingKernel * 0.3 } override protected def k(x: Point[_3D], y: Point[_3D]): DenseMatrix[Double] = faceKernel(x,y) override def domain: Domain[_3D] = RealSpace[_3D] override def outputDim: Int = 3 private def symmetrize(kernel: MatrixValuedPDKernel[_3D]) : MatrixValuedPDKernel[_3D] = { new MatrixValuedPDKernel[_3D] { override def outputDim = 3 override def k(x: Point[_3D], y: Point[_3D]): DenseMatrix[Double] = { val ybar = Point(-y.x, y.y, y.z) val xbar = Point(-x.x, x.y, x.z) val I = DenseMatrix.eye[Double](3) I(0, 0) = 1 val IBar = DenseMatrix.eye[Double](3) IBar(0, 0) = -1 I * kernel(x, y) + IBar * (kernel(x, ybar)) } override def domain: Domain[_3D] = kernel.domain } } } object FaceKernel { case class LevelWithScale(level : Int, scale : Double) } ================================================ FILE: src/main/scala/registration/modelbuilding/FaceMask.scala ================================================ /* * Copyright University of Basel, Graphics and Vision Research Group * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ package registration.modelbuilding import scalismo.common.{PointId, UnstructuredPointsDomain} import scalismo.geometry.{Point, _3D} import scalismo.kernels.GaussianKernel import scalismo.mesh.{ScalarMeshField, TriangleMesh} import scalismo.utils.Memoize case class FaceMask(levelMask: ScalarMeshField[Int], semanticMask: ScalarMeshField[Int]) { def isEarRegion(id : PointId) : Boolean = { semanticMask(id) == 1 } def isLipPoint(id : PointId) : Boolean = { semanticMask(id) == 2 } def isNoseRegion(id : PointId) : Boolean = { semanticMask(id) == 3 } // Returns a value in the interval [0,1] indicating whether a point belongs to the region def computeSmoothedRegions(referenceMesh: TriangleMesh[_3D], level : Int, stddev : Double) : Point[_3D] => Double = { val transformedMask = levelMask.copy(mesh = referenceMesh) val pointsWithRegions = transformedMask.pointsWithValues.toIndexedSeq val regionSmoother = GaussianKernel[_3D](stddev) val regionPts = UnstructuredPointsDomain(pointsWithRegions.filter(_._2 >= level).map(_._1)) def regionWeight(p : Point[_3D]) : Double = { regionSmoother(regionPts.findClosestPoint(p).point,p) } Memoize(regionWeight,referenceMesh.pointSet.numberOfPoints) } } ================================================ FILE: src/main/scala/registration/utils/VisualLogger.scala ================================================ /* * Copyright University of Basel, Graphics and Vision Research Group * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ package registration.utils import java.awt.Color import breeze.linalg.DenseVector import scalismo.geometry._3D import scalismo.mesh.TriangleMesh import scalismo.statisticalmodel.StatisticalMeshModel import scalismo.ui.api._ object VisualLogger { var ui : Option[ScalismoUI] = None//Some(ScalismoUI("Visual Logger")) val modelGroup = ui.map(_.createGroup("Model")) var modelView : Option[StatisticalMeshModelViewControls] = None val targetGroup = ui.map(_.createGroup("Target")) var targetMeshView : Option[TriangleMeshView] = None def showTargetMesh(targetMesh : TriangleMesh[_3D]) : Unit = { remove(targetMeshView) targetMeshView = show(VisualLogger.targetGroup, targetMesh, "target") targetMeshView.map(_.color = Color.RED) } def showStatisticalShapeModel(ssm : StatisticalMeshModel) : Unit = { removeModel(modelView) modelView = show(modelGroup, ssm, "gpmodel") modelView.map(_.meshView.opacity = 0.7) } def updateModelView(coeffs : DenseVector[Double]) : Unit = { if (modelView.isDefined) { modelView.get.shapeModelTransformationView.shapeTransformationView.coefficients = coeffs } } private def show[A](group : Option[Group], t : A, name : String)(implicit sic : ShowInScene[A]): Option[sic.View] = { for { ui <- ui g <- group } yield { ui.show(g, t, name) } } def remove[V <: ObjectView](view : Option[V]): Unit = { view.foreach(_.remove()) } def removeModel(view : Option[StatisticalMeshModelViewControls]): Unit = { for {v <- view} { v.meshView.remove() v.shapeModelTransformationView.remove() } } }