In this step-by-step tutorial, we will cover how to build a Scala REST API using Finatra version 2. Finatra version 2 is a complete rewrite of finatra and is significantly faster(50 times according to documentation) than version 1.x.
> This blog is part of my year long blog series [52 Technologies in 2016](https://github.com/shekhargulati/52-technologies-in-2016)
## Prerequisite
1. Scala 2.11.7
2. IntelliJ Idea Community Edition
3. JDK 8
> Code for today's demo application is on Github at [fitman](fitman)
## Building an application from scratch
In this blog, we will build a simple application called **fitman**. The goal of this application is to track weight of an individual. Every week, a user enter his/her weight and a status message describing how they are feeling. This will allow them to view a timeline of their body weight.
### Step 1: Create a Scala SBT project using IntelliJ Idea
Open IntelliJ Idea and select Scala > SBT project. You will see screen as shown below.
After selecting, press ***Next*** button. Enter the project details and press ***Finish*** button.
This will create a Scala based SBT project that we will use.
> You can use any other IDE or tool as well to scaffold a Scala SBT project.
### Step 2: Adding required dependencies to build.sbt
Your `build.sbt` should look like following:
```scala
name := "fitman"
version := "1.0"
scalaVersion := "2.11.7"
lazy val versions = new {
val finatra = "2.1.2"
val logback = "1.1.3"
}
resolvers ++= Seq(
Resolver.sonatypeRepo("releases"),
"Twitter Maven" at "https://maven.twttr.com"
)
libraryDependencies += "com.twitter.finatra" % "finatra-http_2.11" % versions.finatra
libraryDependencies += "com.twitter.finatra" % "finatra-slf4j_2.11" % versions.finatra
libraryDependencies += "ch.qos.logback" % "logback-classic" % versions.logback
```
Out of three dependencies mentioned above, `finatra-http_2.11` is only required. `finatra-slf4j_2.11` and `logback-classic` are added for logging purpose only.
> Couple of things that disappointed me once I added above mentioned dependencies was 1) time it took to download all the dependencies 2) few transient dependencies like `twitter-metrics` are not present on Maven central so you have to add Twitter's Maven repository located at https://maven.twttr.com.
### Step 3: Fitman says Hello
A finatra app consists of an http server, a list of controllers, and zero or more filters. Let's create a simple Scala class that extends finatra's `com.twitter.finatra.http.HttpServer` as shown below.
```scala
import com.twitter.finatra.http.HttpServer
object FitmanApp extends FitmanServer
class FitmanServer extends HttpServer
```
In the code shown above, we created a server `FitmanServer` that extends `com.twitter.finatra.http.HttpServer`. `HttpServer` extends `TwitterServer` and adds configuration specific to an http server. `TwitterServer` is a template using which other types of servers can be created. `FitmanApp` is an object that is used to launch the server. From the documentation,
> The reason for having a separate object is to allow server to be instantiated multiple times in tests without worrying about static state persisting across test runs in the same JVM.
Also, according to documentation **Finatra convention is to create a Scala object with a name ending in **Main****. I prefer to use convention where name ends with `App` so I am using that.
You can run the application just like you will run any Scala main program. In IntelliJ, right click and click **Fitman App** as shown below.
This will launch the netty based server at port `8888`. As there is no route configured, so you will not be able to do anything useful.
> If you want to use any other port that 8888 then you can use `-http.port` flag and set the it to your preferred value like -http.port=:8080
Let's write our first controller -- `HelloController` in the `FitmanApp.scala` file as shown below.
```scala
import com.twitter.finagle.http.Request
import com.twitter.finatra.http.routing.HttpRouter
import com.twitter.finatra.http.{Controller, HttpServer}
object FitmanApp extends FitmanServer
class FitmanServer extends HttpServer {
override protected def configureHttp(router: HttpRouter): Unit = {
router.add(new HelloController)
}
}
class HelloController extends Controller {
get("/hello") { request: Request =>
"Fitman says hello"
}
}
```
In the code shown above, we created `HelloController` which extends finatra `Controller` abstract class. `HelloController` is defined with one endpoint - `/hello`. When an HTTP GET request is made to `/hello` then the associated callback function will be called. The callback function has `callback: RequestType => ResponseType` signature. It accepts a `com.twitter.finagle.http.Request` and returns back a response of any type that can be converted to `com.twitter.finagle.http.Response`. The callback function in this case just returns a string.
To make server aware of the controller, we registered `HelloController` with `FitmanServer` by overriding its `configureHttp` method. The `configureHttp` exposes `HttpRouter` that is used to register an instance of `HelloController`.
> You can also ask server to handle instantiation of controller by passing the type of controller to the add method instead of its instance. This becomes very useful when used along with dependency injection framework like Guice. We will discuss it later in detail.
```scala
override protected def configureHttp(router: HttpRouter): Unit = {
router.add[HelloController]
}
```
Now, when you make an HTTP GET request to `http://localhost:8888/hello` you will receive fitman ascii art as shown below.
```
→ curl -i http://localhost:8888/hello
HTTP/1.1 200 OK
Content-Length: 17
Fitman says hello
```
#### Admin Interface
Every Finatra by default exposes an admin interface at http://localhost:9990/admin that you can use to get system level details like CPU usage profile, heap profile, server information, and many other. To learn about all admin features refer to [Twitter Server documentation](https://twitter.github.io/twitter-server/Features.html#http-admin-interface).
You can configure admin interface to run on any other port by passing `-admin.port` flag. In IntelliJ, edit your run configuration as shown below.
From now on Admin interface will be available at http://localhost:10000/admin.
#### Overriding default server configuration
There are two ways you can override default server configuration values. One way is to use flags as discussed previously with admin and http ports. The other way you can override default server configuration is by overriding fields in the `FitmanServer` as shown below.
```scala
class FitmanServer extends HttpServer {
override protected def defaultFinatraHttpPort: String = ":8080"
override protected def defaultTracingEnabled: Boolean = false
override protected def defaultHttpServerName: String = "FitMan"
override protected def configureHttp(router: HttpRouter): Unit = {
router.add(new HelloController)
}
}
```
### Step 4: Let's write feature test for HelloController
One of the feature of Finatra that impressed me most was its inbuilt support for feature testing. Feature testing is a form of blackbox testing that tests a particular feature from outside.
Let's add dependencies to `build.sbt` file. You can view full build.sbt [here](https://github.com/shekhargulati/52-technologies-in-2016/blob/master/01-finatra/fitman/build.sbt).
```scala
libraryDependencies += "com.twitter.finatra" % "finatra-http_2.11" % versions.finatra % "test"
libraryDependencies += "com.twitter.inject" % "inject-server_2.11" % versions.finatra % "test"
libraryDependencies += "com.twitter.inject" % "inject-app_2.11" % versions.finatra % "test"
libraryDependencies += "com.twitter.inject" % "inject-core_2.11" % versions.finatra % "test"
libraryDependencies += "com.twitter.inject" %% "inject-modules" % versions.finatra % "test"
libraryDependencies += "com.google.inject.extensions" % "guice-testlib" % versions.guice % "test"
libraryDependencies += "com.twitter.finatra" % "finatra-jackson_2.11" % versions.finatra % "test"
libraryDependencies += "com.twitter.finatra" % "finatra-http_2.11" % versions.finatra % "test" classifier "tests"
libraryDependencies += "com.twitter.inject" % "inject-server_2.11" % versions.finatra % "test" classifier "tests"
libraryDependencies += "com.twitter.inject" % "inject-app_2.11" % versions.finatra % "test" classifier "tests"
libraryDependencies += "com.twitter.inject" % "inject-core_2.11" % versions.finatra % "test" classifier "tests"
libraryDependencies += "com.twitter.inject" % "inject-modules_2.11" % versions.finatra % "test" classifier "tests"
libraryDependencies += "com.google.inject.extensions" % "guice-testlib" % versions.guice % "test" classifier "tests"
libraryDependencies += "com.twitter.finatra" % "finatra-jackson_2.11" % versions.finatra % "test" classifier "tests"
libraryDependencies += "org.scalatest" % "scalatest_2.11" % "2.2.4" % "test"
libraryDependencies += "org.specs2" %% "specs2" % "2.3.12" % "test"
```
Let's write our first feature test that will test the `/hello` endpoint. To create a feature test, you have to extend a trait called `FeatureTest`. You have to provide implementation for server definition as shown below. We created an instance of `EmbeddedHttpServer` passing it our application twitter server -- `FitmanServer`.
```scala
import com.twitter.finagle.http.Status
import com.twitter.finatra.http.test.EmbeddedHttpServer
import com.twitter.inject.server.FeatureTest
class HelloControllerFeatureTest extends FeatureTest {
override val server: EmbeddedHttpServer = new EmbeddedHttpServer(
twitterServer = new FitmanServer)
"Say Hello" in {
server.httpGet(
path = "/hello",
andExpect = Status.Ok,
withBody = "Fitman says hello"
)
}
}
```
This test will start an embedded http server and will make an actual HTTP GET request to the `/hello` endpoint. We asserted that HTTP status code returned by our service is 200 i.e. OK and response body contains text `Fitman says hello`. If you change the body text to something other than `Fitman says hello` then test will fail with detailed message outlining the difference between texts as shown below.
```
"Fitman says hello[]" did not equal "Fitman says hello[123]"
```
This allows you to test the externally visible features of the API.
### Step 5: Let's capture weight
Let's first write the feature test for our WeightResource. The feature test will test that when an HTTP POST request is made to `/weights` endpoint then weight will be stored in some database. In today's blog, we will use a mutable Map to act as a database. Later in this series, we will cover how to work with databases in Scala. We will update our blog then.
```scala
import com.shekhargulati.fitman.FitmanServer
import com.twitter.finagle.http.Status
import com.twitter.finatra.http.test.EmbeddedHttpServer
import com.twitter.inject.server.FeatureTest
class WeightResourceFeatureTest extends FeatureTest {
override val server = new EmbeddedHttpServer(
twitterServer = new FitmanServer
)
"WeightResource" should {
"Save user weight when POST request is made" in {
server.httpPost(
path = "/weights",
postBody =
"""
|{
|"user":"shekhar",
|"weight":85,
|"status":"Feeling great!!!"
|}
""".stripMargin,
andExpect = Status.Created,
withLocation = "/weights/shekhar"
)
}
}
}
```
When you will run the test case then this test will fail as we have not yet added functionality for WeightResource.
Our data model looks like as shown below. It should have same field names as JSON.
```scala
case class Weight(
user: String,
weight: Int,
status: Option[String],
postedAt: Instant = Instant.now()
)
```
Now, let's write our WeightResource.
```scala
import com.twitter.finatra.http.Controller
import scala.collection.mutable
class WeightResource extends Controller {
val db = mutable.Map[String, List[Weight]]()
post("/weights") { weight: Weight =>
val weightsForUser = db.get(weight.user) match {
case Some(weights) => weights :+ weight
case None => List(weight)
}
db.put(weight.user, weightsForUser)
response.created.location(s"/weights/${weight.user}")
}
}
```
Update FitmanServer with new Controller
```
router.add(new WeightResource)
```
In the code shown above, we did the following:
1. We created a mutable Map to store weight for a user.
2. In the `post("/weights")` callback, we are directly using our case class Weight instead of using Finagle request. Finatra automatically converts the request body to the case class.
3. In the post method callback, we first check whether the user exists in the db or not. If user exists, then we add weight to its existing weights collection else we create new List with weight.
4. Finally, we return the response back to the user. `response.created` makes sure that HTTP status 201 is set. We also set the location header to point to a new resource.
### Step 6: View user weight
Now, let's write our second operation that will return all captured weights for a user. We will start with a feature test as shown below.
```scala
"List all weights for a user when GET request is made" in {
val response = server.httpPost(
path = "/weights",
postBody =
"""
|{
|"user":"test_user_1",
|"weight":80,
|"posted_at" : "2016-01-03T14:34:06.871Z"
|}
""".stripMargin,
andExpect = Status.Created
)
server.httpGetJson[List[Weight]](
path = response.location.get,
andExpect = Status.Ok,
withJsonBody =
"""
|[
| {
| "user" : "test_user_1",
| "weight" : 80,
| "posted_at" : "2016-01-03T14:34:06.871Z"
| }
|]
""".stripMargin
)
}
```
Add the following method to `WeightResource`
```scala
get("/weights/:user") { request: Request =>
db.getOrElse(request.params("user"), List())
}
```
### Step 7: Getting logging right
In step 2, we added `finatra-slf4j_2.11` and `logback-classic` dependencies to the classpath so that we can effectively log in our application. `finatra` uses SLF4J api for framework logging. SLF4J provides an API abstraction for various logging framework like log4j, logback, etc. Developers are free to choose their favorite logging library that works with SLF4J. finatra documentation recommends to use Logback as an SLF4J binding as it is much more superior and performant than other logging libraries.
To log in your application, you have to mixin `com.twitter.inject.Logging` trait into your application's object or class. Let's add some log statements to `WeightResource`.
```scala
import com.twitter.finagle.http.Request
import com.twitter.finatra.http.Controller
import com.twitter.inject.Logging
import org.joda.time.Instant
import scala.collection.mutable
class WeightResource extends Controller with Logging {
val db = mutable.Map[String, List[Weight]]()
get("/weights") { request: Request =>
info("finding all weights for all users...")
db
}
get("/weights/:user") { request: Request =>
info( s"""finding weight for user ${request.params("user")}""")
db.getOrElse(request.params("user"), List())
}
post("/weights") { weight: Weight =>
val r = time(s"Total time take to post weight for user '${weight.user}' is %d ms") {
val weightsForUser = db.get(weight.user) match {
case Some(weights) => weights :+ weight
case None => List(weight)
}
db.put(weight.user, weightsForUser)
response.created.location(s"/weights/${weight.user}")
}
r
}
}
case class Weight(
user: String,
weight: Int,
status: Option[String],
postedAt: Instant = Instant.now()
)
```
### Step 8: Validations
Finatra comes with validation annotations that can be used to add validation support. Out of the box Finatra comes with following validation annotations.
1. CountryCode
2. FutureTime
3. Max
4. Min
5. NotEmpty
6. OneOf
7. PastTime
8. Range
9. Size
10. TimeGranularity
11. UUID
All these annotations are defined in `finatra-jackson_2.11` module so you have to add that to build.sbt
```scala
libraryDependencies += "com.twitter.finatra" % "finatra-jackson_2.11" % versions.finatra
```
Let's write a test case
```scala
"Bad request when user is not present in request" in {
server.httpPost(
path = "/weights",
postBody =
"""
|{
|"weight":85
|}
""".stripMargin,
andExpect = Status.BadRequest
)
}
```
If you run the test now, it will fail with Http status code 500 i.e. Internal server error.
To make it work first we have to register a filter in the FitmanServer.
```scala
import com.twitter.finatra.http.filters.CommonFilters
class FitmanServer extends HttpServer {
override protected def configureHttp(router: HttpRouter): Unit = {
router
.filter[CommonFilters]
.add[HelloController]
.add[WeightResource]
}
}
```
Now test will pass.
```scala
"Bad request when data not in range" in {
server.httpPost(
path = "/weights",
postBody =
"""
|{
|"user":"testing12345678910908980898978798797979789",
|"weight":250
|}
""".stripMargin,
andExpect = Status.BadRequest,
withErrors = Seq(
"user: size [42] is not between 1 and 25",
"weight: [250] is not between 25 and 200"
)
)
}
```
Update the `Weight` case class
```scala
case class Weight(
@Size(min = 1, max = 25) user: String,
@Range(min = 25, max = 200) weight: Int,
status: Option[String],
postedAt: Instant = Instant.now()
)
```
That's it for this week. Please give your feedback [https://github.com/shekhargulati/52-technologies-in-2016/issues/1](https://github.com/shekhargulati/52-technologies-in-2016/issues/1)
[](https://github.com/igrigorik/ga-beacon)
================================================
FILE: 01-finatra/fitman/.gitignore
================================================
# Created by .ignore support plugin (hsz.mobi)
### Scala template
*.class
*.log
# sbt specific
.cache
.history
.lib/
dist/*
target/
lib_managed/
src_managed/
project/boot/
project/plugins/project/
# Scala-IDE specific
.scala_dependencies
.worksheet
### SBT template
# Simple Build Tool
# http://www.scala-sbt.org/release/docs/Getting-Started/Directories.html#configuring-version-control
### JetBrains template
# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio
*.iml
## Directory-based project format:
.idea/
# if you remove the above rule, at least ignore the following:
# User-specific stuff:
# .idea/workspace.xml
# .idea/tasks.xml
# .idea/dictionaries
# Sensitive or high-churn files:
# .idea/dataSources.ids
# .idea/dataSources.xml
# .idea/sqlDataSources.xml
# .idea/dynamic.xml
# .idea/uiDesigner.xml
# Gradle:
# .idea/gradle.xml
# .idea/libraries
# Mongo Explorer plugin:
# .idea/mongoSettings.xml
## File-based project format:
*.ipr
*.iws
## Plugin-specific files:
# IntelliJ
/out/
# mpeltonen/sbt-idea plugin
.idea_modules/
# JIRA plugin
atlassian-ide-plugin.xml
# Crashlytics plugin (for Android Studio and IntelliJ)
com_crashlytics_export_strings.xml
crashlytics.properties
crashlytics-build.properties
================================================
FILE: 01-finatra/fitman/build.sbt
================================================
name := "fitman"
version := "1.0"
scalaVersion := "2.11.7"
lazy val versions = new {
val finatra = "2.1.2"
val logback = "1.1.3"
val guice = "4.0"
}
resolvers ++= Seq(
Resolver.sonatypeRepo("releases"),
"Twitter Maven" at "https://maven.twttr.com"
)
libraryDependencies += "com.twitter.finatra" % "finatra-http_2.11" % versions.finatra
libraryDependencies += "com.twitter.finatra" % "finatra-jackson_2.11" % versions.finatra
libraryDependencies += "com.twitter.finatra" % "finatra-slf4j_2.11" % versions.finatra
libraryDependencies += "ch.qos.logback" % "logback-classic" % versions.logback
libraryDependencies += "com.twitter.finatra" % "finatra-http_2.11" % versions.finatra % "test"
libraryDependencies += "com.twitter.inject" % "inject-server_2.11" % versions.finatra % "test"
libraryDependencies += "com.twitter.inject" % "inject-app_2.11" % versions.finatra % "test"
libraryDependencies += "com.twitter.inject" % "inject-core_2.11" % versions.finatra % "test"
libraryDependencies += "com.twitter.inject" %% "inject-modules" % versions.finatra % "test"
libraryDependencies += "com.google.inject.extensions" % "guice-testlib" % versions.guice % "test"
libraryDependencies += "com.twitter.finatra" % "finatra-jackson_2.11" % versions.finatra % "test"
libraryDependencies += "com.twitter.finatra" % "finatra-http_2.11" % versions.finatra % "test" classifier "tests"
libraryDependencies += "com.twitter.inject" % "inject-server_2.11" % versions.finatra % "test" classifier "tests"
libraryDependencies += "com.twitter.inject" % "inject-app_2.11" % versions.finatra % "test" classifier "tests"
libraryDependencies += "com.twitter.inject" % "inject-core_2.11" % versions.finatra % "test" classifier "tests"
libraryDependencies += "com.twitter.inject" % "inject-modules_2.11" % versions.finatra % "test" classifier "tests"
libraryDependencies += "com.google.inject.extensions" % "guice-testlib" % versions.guice % "test" classifier "tests"
libraryDependencies += "com.twitter.finatra" % "finatra-jackson_2.11" % versions.finatra % "test" classifier "tests"
libraryDependencies += "org.scalatest" % "scalatest_2.11" % "2.2.4" % "test"
libraryDependencies += "org.specs2" %% "specs2" % "2.3.12" % "test"
================================================
FILE: 01-finatra/fitman/project/build.properties
================================================
sbt.version = 0.13.8
================================================
FILE: 01-finatra/fitman/project/plugins.sbt
================================================
logLevel := Level.Warn
================================================
FILE: 01-finatra/fitman/src/main/resources/logback.xml
================================================
## Install sbt on your machine
If you are on mac then, you can use package manager like `brew` to install `sbt` on your machine:
```bash
$ brew install sbt
```
For other systems, download the latest version of sbt from the [website](http://www.scala-sbt.org/download.html). Current production version is `0.13.16`. You can download by clicking [https://cocl.us/sbt01316zip](https://cocl.us/sbt01316zip).
You can refer to manual instructions from `sbt` website http://www.scala-sbt.org/0.13/tutorial/Manual-Installation.html. This blog is written using sbt version `0.13.9`.
Once you have successfully installed `sbt` on your machine, create an empty directory somewhere in your file system which we will use for this blog.
To view the basic information about `sbt`, we can use `about` task. The `sbt about` task and its output is shown below:
```
$ sbt about
[info] Set current project to code (in build file:/Users/shekhargulati/blogs/sbt-playground)
[info] This is sbt 0.13.9
[info] The current project is {file:/Users/shekhargulati/blogs/sbt-playground}code 0.1-SNAPSHOT
[info] The current project is built against Scala 2.10.5
[info] Available Plugins: sbt.plugins.IvyPlugin, sbt.plugins.JvmPlugin, sbt.plugins.CorePlugin, sbt.plugins.JUnitXmlReportPlugin
[info] sbt, sbt plugins, and build definitions are using Scala 2.10.5
```
As you can see in the second line of the output we are using sbt version 0.13.9.
> First time you run `sbt`, it will download some jars that are required by sbt to perform its job. The default sbt installation is very minimalistic and does not come bundled with everything.
## Getting started with sbt
`sbt` terminology consists of two terms -- **tasks** and **settings**. A task defines an action which you want to perform like compile. A setting is used to define a value for example name and version of the project.
With `sbt` whenever you want to perform any action you execute a task. Task is the unit of currency in `sbt`. A task can depend on another task to do its job. `sbt` creates a task dependency graph to determine which task should run first. If task `t1` depends on task `t2` then task `t2` will be executed first and then task `t1` will be executed. You can view all the tasks applicable to a project by running `sbt tasks` task:
```bash
$ sbt tasks
```
The above task will produce the following output:
```
[info] Set current project to code (in build file:/Users/shekhargulati/blogs/tasky/)
This is a list of tasks defined for the current project.
It does not list the scopes the tasks are defined in; use the 'inspect' task for that.
Tasks produce values. Use the 'show' task to run the task and print the resulting value.
clean Deletes files produced by the build, such as generated sources, compiled classes, and task caches.
compile Compiles sources.
console Starts the Scala interpreter with the project classes on the classpath.
consoleProject Starts the Scala interpreter with the sbt and the build definition on the classpath and useful imports.
consoleQuick Starts the Scala interpreter with the project dependencies on the classpath.
copyResources Copies resources to the output directory.
doc Generates API documentation.
package Produces the main artifact, such as a binary jar. This is typically an alias for the task that actually does the packaging.
packageBin Produces a main artifact, such as a binary jar.
packageDoc Produces a documentation artifact, such as a jar containing API documentation.
packageSrc Produces a source artifact, such as a jar containing sources and resources.
publish Publishes artifacts to a repository.
publishLocal Publishes artifacts to the local Ivy repository.
publishM2 Publishes artifacts to the local Maven repository.
run Runs a main class, passing along arguments provided on the task line.
runMain Runs the main class selected by the first argument, passing the remaining arguments to the main method.
test Executes all tests.
testOnly Executes the tests provided as arguments or all tests if no arguments are provided.
testQuick Executes the tests that either failed before, were not run or whose transitive dependencies changed, among those provided as arguments.
update Resolves and optionally retrieves dependencies, producing a report.
```
> One thing that surprised me about `sbt` is that it runs all the tasks in parallel by default. If the tasks have dependencies, then `sbt` uses the task dependency graph to determine which tasks can run in parallel and which can run in a sequential manner. To make it clear, let's suppose we have three tasks `t1`, `t2`, and `t3`. `t1` and `t3` depends on `t2` then `t2` will run first and `t1` and `t3` will run in parallel.
## sbt modes
You can use `sbt` in two modes -- command-line mode and interactive mode. In the command-line mode, you run `sbt` task from your machine terminal. Once the task successfully finishes then `sbt` exits. For example, when you ran `sbt about` task, it printed `sbt` and build information on the console and then `sbt` exited and you were back to your terminal. In the interactive mode, you run `sbt` command and it launches a `sbt` shell. Inside the `sbt` shell session, you run `sbt` tasks.
## Create an sbt Scala project
In this tutorial, we will build a simple task management app -- `tasky`. A task management app will allow us to work with our daily to-do items. Create a new directory `tasky` at any convenient location on your filesystem. Once created, change directory to `tasky`:
```bash
$ mkdir tasky
$ cd tasky
```
> Code for demo application is on [GitHub](./tasky)
Inside the `tasky` directory, create a new file -- `build.sbt` to house the build script. `build.sbt` is the name of the sbt build script. The content of `build.sbt` is shown below:
```scala
name := "tasky"
version := "0.1.0"
```
`:=` is a function defined in the `sbt` library. It is used to define a setting that overwrites any previous value without referring to other settings. For example, `name := "tasky"` will overwrite any previous value set in the `name` variable.
Now, run the `sbt` command:
```
$ sbt
[info] Set current project to tasky (in build file:/Users/shekhargulati/blogs/tasky)
>
```
Once you are inside the `sbt` shell, you can run various `sbt` tasks. To view all the tasks available you can use `help` task:
```bash
> help
```
```
help Displays this help message or prints detailed help on requested tasks (run 'help {{ .Title }}
{{ .Summary }}
Benefits of using slick:
1. Type safety and compile time checking
2. Generate query for any database
3. Composable
4. Back-pressure built-in
5. Streaming support via reactive streams
6. You can use SQL as well
From the [Slick docs](http://slick.typesafe.com/doc/3.1.1/introduction.html#functional-relational-mapping):
> **The language integrated query model in Slick’s FRM is inspired by the LINQ project at Microsoft and leverages concepts tracing all the way back to the early work of Mnesia at Ericsson.**
Slick supports most of the relational databases in the market. You can view full list [here](http://slick.typesafe.com/doc/3.1.1/supported-databases.html). You can work with all open source databases like MySQL, PostgreSQL for free. Databases like Oracle, SQL Server, and DB2 are available as closed extensions that you can use only after buying subscription.
> **This blog is part of my year long blog series [52 Technologies in 2016](https://github.com/shekhargulati/52-technologies-in-2016)**
## Github repository
The code for today’s demo application is available on github: [tasky](./tasky).
## Getting Started
Create a new directory `tasky` on your filesystem. Inside the `tasky` directory create a sbt build file `build.sbt` with the following contents.
```scala
name := "tasky"
description := "A simple task manager for humans"
version := "0.1.0"
scalaVersion := "2.11.7"
libraryDependencies += "com.typesafe.slick" %% "slick" % "3.1.1"
libraryDependencies += "com.h2database" % "h2" % "1.4.191"
libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.1.3"
libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6" % "test"
```
> **In this tutorial, we will Slick version 3.1.1**
In the `build.sbt` file shown above, we have first defined basic information about the project like name, version, and description. We have also specified that we are going to use Scala version `2.11.7`. After that we have declared few dependencies. The only required dependency is of `Slick`. `logback` is used for logging and `scalatest` will be used for writing test cases. In this tutorial, we will use `h2` database so we have declared its dependency as well. `h2` is an in-memory SQL database implementation written in `Java`. It runs in the same process as your application and is useful for testing and getting started purposes. For real apps, you should use databases like MySQL or PostgreSQL.
Create the following directory structure inside the `tasky` directory.
```bash
$ mkdir -p src/main/scala
$ mkdir -p src/test/scala
```
Now, we have a basic Scala SBT project setup for Slick application development.
Next, import the project in your favorite IDE.
## Define Database Tables
Tables represent mapping between Scala datatypes and database tables. Create a new package `datamodel` inside the `src/main/scala` directory.
Inside the `datamodel` package, create a scala object `DataModel.scala`.
```scala
package datamodel
import slick.driver.H2Driver.api._
object DataModel {
}
```
The import `slick.driver.H2Driver.api._` is required to tell which Slick database API we will use in our application. As shown above, we are using H2 for our application.
Let's create a new Scala datatype for our task management application. To keep things simple and easy to understand, we will start with only one domain object i.e. Task. `Task` case class is shown below.
```scala
import java.time.LocalDateTime
object DataModel {
case class Task(
title: String,
description: String = "",
createdAt: LocalDateTime = LocalDateTime.now(),
dueBy: LocalDateTime,
tags: Set[String] = Set(),
id: Long = 0L)
}
```
The case class represent a `Task` datatype with six fields. This will map to a task table that will store a list of tasks that a user has to perform. As you can see, we have used different datatypes like String, Java 8 LocalDateTime, Set, and Long. LocalDateTime is part of Java 8 Date Time API. We have also given default values to some of these fields. This will allow us to not pass these value when we are constructing task objects. So, we can create a task by just providing `title` and `dueBy` values.
> **Please refer to [my Java 8 tutorial](https://github.com/shekhargulati/java8-the-missing-tutorial/blob/master/08-date-time-api.md) if you are new to Java 8**
Now let's create a table mapping for our Task case class.
```scala
object DataModel {
case class Task(
title: String,
description: String = "",
createdAt: LocalDateTime = LocalDateTime.now(),
dueBy: LocalDateTime,
tags: Set[String] = Set(),
id: Long = 0L)
class TaskTable(tag: Tag) extends Table[Task](tag, "tasks") {
def title = column[String]("title")
def description = column[String]("description")
def createdAt = column[LocalDateTime]("createdAt")
def dueBy = column[LocalDateTime]("dueBy")
def tags = column[Set[String]]("tags")
def id = column[Long]("id", O.PrimaryKey, O.AutoInc)
override def * = (title, description, createdAt, dueBy, tags, id) <>(Task.tupled, Task.unapply)
}
}
```
Let's understand the `TaksTable` class code shown above.
1. Every table needs to extend `Table` abstract class. Table class needs a type parameter that tells what we will store in our table. Here, we are storing `Task` in the TaskTable. TaskTable constructors needs two mandatory fields - tag and table name. As shown above, we have used `tasks` as the name of our table. `tag` is something internal to slick that you have to pass to the `Table` constructor. This is used by slick to determine shape of a single table row. There is not much mentioned about `tag` in the slick documentation so I might not be 100% correct.
2. Next, we defined definitions of each of the columns. These map one-to-one to our domain class `Task`.
3. `id` is our primary key. In the column definition, we have said slick to make id an auto incrementing primary key. This will make sure database allocate id to each row in auto increment manner.
4. The `*` method is the default projection of our table. You have to define this method in your `Table` class. The type of the `*` projection has to be the same as type specified in the `Table` type parameter. In our case, both have to be `Task`. The `<>` method is used to convert between a tuple `(title, description, createdAt, dueBy, tags, id)` and `Task` data type. The `<>` needs two functions - first takes a tuple and convert it to an object and second a function that converts an object to a tuple.
> It is not required to use a case class you could have also used a regular Scala class as well. If you do use a regular class, then you have to provide two extra functions corresponding to `tupled` and `unapply`. The advantage that we get by using a case class is that it provides `tupled` and `unapply` methods. In the code shown below, we have created a Task object and defined two methods `toTask` and `fromTask`. These methods will serve the purpose of `tupled` and `unapply` methods.
```scala
class Task(
val title: String,
val description: String = "",
val createdAt: LocalDateTime = LocalDateTime.now(),
val dueBy: LocalDateTime,
val tags: Set[String] = Set[String](),
val id: Long = 0L)
object Task {
def apply(title: String,
description: String = "",
createdAt: LocalDateTime = LocalDateTime.now(),
dueBy: LocalDateTime,
tags: Set[String] = Set[String](),
id: Long = 0L): Task = new Task(title, description, createdAt, dueBy, tags, id)
def toTask(t: (String, String, LocalDateTime, LocalDateTime, Set[String], Long)): Task = new Task(t._1, t._2, t._3, t._4, t._5, t._6)
def fromTask(task: Task): Option[(String, String, LocalDateTime, LocalDateTime, Set[String], Long)] = Some((task.title, task.description, task.createdAt, task.dueBy, task.tags, task.id))
}
```
## Create TableQuery object
Once we have defined our table definition `TaskTable`, we have to define a value of type `TableQuery` which represents an actual database table. It provides a query DSL that you can use to interact with the table.
```scala
lazy val Tasks = TableQuery[TaskTable]
```
## Define Custom Mapping
If you try to compile the code that we have written so far it will not compile. The reason for that is slick does not support Java 8 `LocalDateTime` and `Set[String]` datatypes for column definition. However, we can write our custom mappers that will convert our types to the type `Slick` understands. Create a new object `ColumnDataMapper` in the same file `DataModel.scala` as shown below.
```scala
object ColumnDataMapper {
implicit val localDateTimeColumnType = MappedColumnType.base[LocalDateTime, Timestamp](
ldt => Timestamp.valueOf(ldt),
t => t.toLocalDateTime
)
implicit val setStringColumnType = MappedColumnType.base[Set[String], String](
tags => tags.mkString(","),
tagsString => tagsString.split(",").toSet
)
}
```
In the code shown above, we have defined two mapper -- a) converts between `LocalDateTime` to `java.sql.Timestamp` and vice-versa b) converts between `Set[String]` to `String` and vice-versa.
Now, add the import for your custom data mappings. You have to explicitly add the custom mappers to the column definition.
```scala
import datamodel.ColumnDataMapper.{localDateTimeColumnType, setStringColumnType}
class TaskTable(tag: Tag) extends Table[Task](tag, "tasks") {
def title = column[String]("title")
def description = column[String]("description")
def createdAt = column[LocalDateTime]("createdAt")(localDateTimeColumnType)
def dueBy = column[LocalDateTime]("dueBy")(localDateTimeColumnType)
def tags = column[Set[String]]("tags")(setStringColumnType)
def id = column[Long]("id", O.PrimaryKey, O.AutoInc)
override def * = (title, description, createdAt, dueBy, tags, id) <>(Task.tupled, Task.unapply)
}
```
Now, code will compile successfully. You can use `sbt compile` task to compile the application.
## Define Schema Create Action
Action represents commands that we want to run against database. Let's write our first action that will create the database schema.
```scala
lazy val Tasks = TableQuery[TaskTable]
val createTaskTableAction = Tasks.schema.create
```
The `createTaskTableAction` action will create the schema when it is executed against database. **Defining an action does not execute it**.
## Execute Schema Create Action
Actions are executed against a database. Slick provides a `Database` type that allows our code to interact with the database. It is a handle to a specific database. To get the handle to a database, you use the following code.
```scala
val db = Database.forConfig("taskydb")
```
The `taskydb` is a reference to a configuration object defined using typesafe config project.
Let's write our first test case that will use the database object to create the schema. In the `src/test/scala`, create a new package `datamodel`. Create a new Scala class `CreateDatabaseSpec` as shown below.
```scala
package datamodel
import org.scalatest.{FunSpec, Matchers}
import slick.driver.H2Driver.api._
import scala.concurrent._
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
class CreateDatabaseSpec extends FunSpec with Matchers {
describe("DataModel Spec") {
it("should create database") {
val db = Database.forConfig("taskydb")
val result = Await.result(db.run(DataModel.createTaskTableAction), 2 seconds)
println(result)
}
}
}
```
In the code shown above:
1. The `scala.concurrent` set of imports are required to tell slick that we will use ExecutionContext defined by the import to execute slick code. We have to do this because slick API is fully asynchronous and executes database calls in a separate thread pool.
2. Then we created our database object using the `taskydb` configuration. This gives us the handle to interact with database.
3. The db object has a method called `run` that executes an action and returns a `Future`. As slick is async in nature, we have wrapped the future in a `Await.result` call to make it easy to test.
You will have to create a file called `application.conf` in the `src/test/resources` directory. Populate it with content shown below.
```
taskydb = {
connectionPool = disabled
url = "jdbc:h2:mem:taskydb"
driver = "org.h2.Driver"
keepAliveConnection = true
}
```
When you will run this code, you will see in the logs that it has create a database schema.
```
18:50:34.955 [ScalaTest-run-running-CreateDatabaseSpec] DEBUG s.backend.DatabaseComponent.action - #1: schema.create [create table "tasks" ("title" VARCHAR NOT NULL,"description" VARCHAR NOT NULL,"createdAt" TIMESTAMP NOT NULL,"dueBy" TIMESTAMP NOT NULL,"tags" VARCHAR NOT NULL,"id" BIGINT GENERATED BY DEFAULT AS IDENTITY(START WITH 1) NOT NULL PRIMARY KEY)]
18:50:35.012 [taskydb-1] DEBUG slick.jdbc.JdbcBackend.statement - Preparing statement: create table "tasks" ("title" VARCHAR NOT NULL,"description" VARCHAR NOT NULL,"createdAt" TIMESTAMP NOT NULL,"dueBy" TIMESTAMP NOT NULL,"tags" VARCHAR NOT NULL,"id" BIGINT GENERATED BY DEFAULT AS IDENTITY(START WITH 1) NOT NULL PRIMARY KEY)
18:50:35.268 [taskydb-1] DEBUG slick.jdbc.JdbcBackend.benchmark - Execution of prepared statement took 23ms
```
On every run of our test case, we will create a new database.
## Insert tasks into Task table
Let's now write a test case that will insert some records into the `Task` table.
```scala
it("should insert single task into database") {
val db = Database.forConfig("taskydb")
val result = Await.result(db.run(DataModel.insertTaskAction(Task(title = "Learn Slick", dueBy = LocalDateTime.now().plusDays(1)))), 2 seconds)
result should be(Some(1))
}
```
The test case shown above calls the `insertTaskAction` passing it a `Task`. The result of `insertTaskAction` is the number of rows affected by the action. As we are only passing one task so we should expect one as result.
Now, let's look at the `insertTaskAction` definition in the `DataModel` object.
```
def insertTaskAction(tasks: Task*) = Tasks ++= tasks.toSeq
```
The insertTaskAction takes a `varargs` of tasks allowing user to pass one or more tasks. To insert tasks, we used `++=` method. According to [slick documentation](http://slick.typesafe.com/doc/3.0.0/queries.html#inserting),
> **`++=` gives you an accumulated count in an Option (which can be None if the database system does not provide counts for all rows)**
## Query table
Let's query the database to select all the tasks in the database.
```scala
it("should list all tasks in the database") {
val tasks = Seq(
Task(title = "Learn Slick", dueBy = LocalDateTime.now().plusDays(1)),
Task(title = "Write blog on Slick", dueBy = LocalDateTime.now().plusDays(2)),
Task(title = "Build a simple application using Slick", dueBy = LocalDateTime.now().plusDays(3))
)
Await.result(db.run(DataModel.insertTaskAction(tasks: _*)), 2 seconds)
val result = Await.result(db.run(DataModel.listTasksAction), 2 seconds)
result should have length 3
}
```
The test case shown above queries the database using `listTasksAction` shown below.
```scala
val listTasksAction = Tasks.result
```
The `listTasksAction` makes a `select "title", "description", "createdAt", "dueBy", "tags", "id" from "tasks"` sql query using the default `*` projection.
## Conclusion
Slick is a powerful library to interact with relational databases. Today, we have just scratched the surface of this feature rich library. You leant how to define table definition, insert data, perform `select *` query. I will write couple more blogs on Slick to cover it in more details. So stay tuned!
That's all for this week. Please provide your valuable feedback by adding a comment to [https://github.com/shekhargulati/52-technologies-in-2016/issues/6](https://github.com/shekhargulati/52-technologies-in-2016/issues/6).
[](https://github.com/igrigorik/ga-beacon)
================================================
FILE: 04-slick/tasky/.gitignore
================================================
# Created by .ignore support plugin (hsz.mobi)
### SBT template
# Simple Build Tool
# http://www.scala-sbt.org/release/docs/Getting-Started/Directories.html#configuring-version-control
target/
lib_managed/
src_managed/
project/boot/
.history
.cache
### JetBrains template
# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio
*.iml
## Directory-based project format:
.idea/
# if you remove the above rule, at least ignore the following:
# User-specific stuff:
# .idea/workspace.xml
# .idea/tasks.xml
# .idea/dictionaries
# Sensitive or high-churn files:
# .idea/dataSources.ids
# .idea/dataSources.xml
# .idea/sqlDataSources.xml
# .idea/dynamic.xml
# .idea/uiDesigner.xml
# Gradle:
# .idea/gradle.xml
# .idea/libraries
# Mongo Explorer plugin:
# .idea/mongoSettings.xml
## File-based project format:
*.ipr
*.iws
## Plugin-specific files:
# IntelliJ
/out/
# mpeltonen/sbt-idea plugin
.idea_modules/
# JIRA plugin
atlassian-ide-plugin.xml
# Crashlytics plugin (for Android Studio and IntelliJ)
com_crashlytics_export_strings.xml
crashlytics.properties
crashlytics-build.properties
### Scala template
*.class
*.log
# sbt specific
.cache
.history
.lib/
dist/*
target/
lib_managed/
src_managed/
project/boot/
project/plugins/project/
# Scala-IDE specific
.scala_dependencies
.worksheet
================================================
FILE: 04-slick/tasky/build.sbt
================================================
name := "tasky"
description := "A simple task manager for humans"
version := "0.1.0"
scalaVersion := "2.11.7"
libraryDependencies += "com.typesafe.slick" %% "slick" % "3.1.1"
libraryDependencies += "com.h2database" % "h2" % "1.4.191"
libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.1.3"
libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6" % "test"
================================================
FILE: 04-slick/tasky/project/plugins.sbt
================================================
addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.8.1")
================================================
FILE: 04-slick/tasky/src/main/scala/datamodel/DataModel.scala
================================================
package datamodel
import java.sql.Timestamp
import java.time.LocalDateTime
import datamodel.ColumnDataMapper._
import slick.driver.H2Driver.api._
object DataModel {
case class Task(
title: String,
description: String = "",
createdAt: LocalDateTime = LocalDateTime.now(),
dueBy: LocalDateTime,
tags: Set[String] = Set[String](),
id: Long = 0L)
class TaskTable(tag: Tag) extends Table[Task](tag, "tasks") {
def title = column[String]("title")
def description = column[String]("description")
def createdAt = column[LocalDateTime]("createdAt")(localDateTimeColumnType)
def dueBy = column[LocalDateTime]("dueBy")(localDateTimeColumnType)
def tags = column[Set[String]]("tags")(setStringColumnType)
def id = column[Long]("id", O.PrimaryKey, O.AutoInc)
override def * = (title, description, createdAt, dueBy, tags, id) <>(Task.tupled, Task.unapply)
}
lazy val Tasks = TableQuery[TaskTable]
val createTaskTableAction = Tasks.schema.create
def insertTaskAction(tasks: Task*) = Tasks ++= tasks.toSeq
val listTasksAction = Tasks.result
}
object ColumnDataMapper {
implicit val localDateTimeColumnType = MappedColumnType.base[LocalDateTime, Timestamp](
ldt => Timestamp.valueOf(ldt),
t => t.toLocalDateTime
)
implicit val setStringColumnType = MappedColumnType.base[Set[String], String](
tags => tags.mkString(","),
tagsString => tagsString.split(",").toSet
)
}
================================================
FILE: 04-slick/tasky/src/test/resources/application.conf
================================================
taskydb = {
connectionPool = disabled
url = "jdbc:h2:mem:taskydb"
driver = "org.h2.Driver"
keepAliveConnection = true
}
================================================
FILE: 04-slick/tasky/src/test/scala/datamodel/DataModelSpec.scala
================================================
package datamodel
import java.time.LocalDateTime
import datamodel.DataModel.Task
import org.scalatest.{BeforeAndAfterEach, FunSpec, Matchers}
import slick.driver.H2Driver.api._
import scala.concurrent._
import scala.concurrent.duration._
class DataModelSpec extends FunSpec with Matchers with BeforeAndAfterEach {
var db: Database = _
override protected def beforeEach(): Unit = {
db = Database.forConfig("taskydb")
Await.result(db.run(DataModel.createTaskTableAction), 2 seconds)
}
override protected def afterEach(): Unit = Await.result(db.shutdown, 2 seconds)
describe("DataModel Spec") {
it("should insert single task into database") {
val result = Await.result(db.run(DataModel.insertTaskAction(Task(title = "Learn Slick", dueBy = LocalDateTime.now().plusDays(1)))), 2 seconds)
result should be(Some(1))
}
it("should insert multiple tasks into database") {
val tasks = Seq(
Task(title = "Learn Slick", dueBy = LocalDateTime.now().plusDays(1)),
Task(title = "Write blog on Slick", dueBy = LocalDateTime.now().plusDays(2)),
Task(title = "Build a simple application using Slick", dueBy = LocalDateTime.now().plusDays(3))
)
val result = Await.result(db.run(DataModel.insertTaskAction(tasks: _*)), 2 seconds)
result should be(Some(3))
}
it("should list all tasks in the database") {
val tasks = Seq(
Task(title = "Learn Slick", dueBy = LocalDateTime.now().plusDays(1)),
Task(title = "Write blog on Slick", dueBy = LocalDateTime.now().plusDays(2)),
Task(title = "Build a simple application using Slick", dueBy = LocalDateTime.now().plusDays(3))
)
Await.result(db.run(DataModel.insertTaskAction(tasks: _*)), 2 seconds)
val result = Await.result(db.run(DataModel.listTasksAction), 2 seconds)
result should have length 3
}
}
}
================================================
FILE: 05-slick/README.md
================================================
Slick 3: Functional Relational Mapping for Mere Mortals Part 2: Querying data
----
Last week we learnt the [basics of Slick](https://github.com/shekhargulati/52-technologies-in-2016/tree/master/04-slick) library. We started with a general introduction of Slick, then covered how to define a table definition, custom mappers, and perform insert queries. Today, we will learn how to perform `select` queries with Slick. Slick allows you to work with database tables in the same way as you work with Scala collections. This means that you can use methods like `map`, `filter`, `sort`, etc. to process data in your table.
> **In case you are new to Slick, please first read [part 1 of Slick tutorial](https://github.com/shekhargulati/52-technologies-in-2016/tree/master/04-slick). This blog is part of my year long blog series [52 Technologies in 2016](https://github.com/shekhargulati/52-technologies-in-2016)**
## Github repository
The code for today’s demo application is available on github: [tasky](./tasky).
## Let's (again) look at the data model
Before we start with querying data, let's again look at the data model. I have added one more field to the `TaskTable`. The field that we have added is an enum to store priority of the task. Enums are useful when a variable can have one of the small set of possible values. In our example application, `Priority` is an enum that can be either `HIGH`, `LOW`, or `MEDIUM`. To create a new enum, create an object that extends `scala.Enumeration` as shown below. We have created `Priority` enum in a new file `Priority.scala` inside the `datamodel` package.
```scala
package datamodel
object Priority extends Enumeration {
type Priority = Value
val HIGH = Value(3)
val MEDIUM = Value(2)
val LOW = Value(1)
}
```
As you can see above, we have provided int values to each enum constant.
After creating our new enum, we have to add its declaration in our `Task` case class as well as `TaskTable`.
```scala
import datamodel.columnDataMappers._
case class Task(
title: String,
description: String = "",
createdAt: LocalDateTime = LocalDateTime.now(),
dueBy: LocalDateTime,
tags: Set[String] = Set[String](),
priority: Priority = Priority.LOW,
id: Long = 0L)
class TaskTable(tag: Tag) extends Table[Task](tag, "tasks") {
def title = column[String]("title")
def description = column[String]("description")
def createdAt = column[LocalDateTime]("createdAt")
def dueBy = column[LocalDateTime]("dueBy")
def tags = column[Set[String]]("tags")
def priority = column[Priority]("priority")
def id = column[Long]("id", O.PrimaryKey, O.AutoInc)
override def * = (title, description, createdAt, dueBy, tags, priority, id) <>(Task.tupled, Task.unapply)
}
```
If we try to compile code now, it will not compile. We have to add column mapping to convert between `Priority` enum to `Int`. This is shown below.
```scala
implicit val priorityMapper = MappedColumnType.base[Priority, Int](
p => p.id,
v => Priority(v)
)
```
Compile and run the test cases using `sbt test` and everything should work fine.
## Select all the tasks in the database
Let's start with the simplest select query i.e. `select * from tasks`. We want to list all the tasks in our database. As discussed last week, we have to create an instance of `TableQuery` that will give us the handle to Slick Query DSL API. We already have instance of `TableQuery` created inside the `dataModels.scala`.
```scala
lazy val Tasks = TableQuery[TaskTable]
```
Create a new Scala object `queries` inside the `queries` package. This object will house all the queries.
```scala
package queries
import datamodel.columnDataMappers._
import datamodel.dataModel.Tasks
import slick.driver.H2Driver.api._
object queries {
}
```
As shown above, we have created a new Scala object `queries` and added the required imports.
1. `import datamodel.columnDataMappers._` is required so that Slick knows how to handle our custom data types like `LocalDateTime`, `Set[String]`, and `Priority`.
2. `import datamodel.dataModel.Tasks` is required so that we can work with the `Tasks` `TableQuery` object.
3. `import slick.driver.H2Driver.api._` is required to tell which Slick database API we will use in our application.
Before we will write query for listing all the tasks in the database let's write a test case. Create a new test specification `QueriesSpec` and populate it with following contents.
```scala
package queries
import java.time.LocalDateTime
import datamodel.dataModel.Task
import datamodel.{Priority, dataModel}
import org.scalatest.{BeforeAndAfterAll, FunSpec, Matchers}
import queries._
import slick.driver.H2Driver.api._
import scala.concurrent._
import scala.concurrent.duration._
class QueriesSpec extends FunSpec with Matchers with BeforeAndAfterAll {
var db: Database = _
var t1: Task = _
var t2: Task = _
var t3: Task = _
var t4: Task = _
var t5: Task = _
var t6: Task = _
var t7: Task = _
override protected def beforeAll(): Unit = {
db = Database.forConfig("taskydb")
Await.result(db.run(dataModel.createTaskTableAction), 2 seconds)
t1 = Task(title = "Write part 1 blog on Slick", dueBy = LocalDateTime.now().minusDays(7), tags = Set("blogging", "scala", "slick"), priority = Priority.HIGH)
t2 = Task(title = "Give a Java 8 training", dueBy = LocalDateTime.now().minusDays(3), tags = Set("java", "training", "travel"), priority = Priority.LOW)
t3 = Task(title = "Write part 2 blog on Slick queries", dueBy = LocalDateTime.now(), tags = Set("blogging", "scala", "slick"), priority = Priority.HIGH)
t4 = Task(title = "Read Good to Great book", dueBy = LocalDateTime.now().plusDays(15), tags = Set("reading", "books", "startup"), priority = Priority.MEDIUM)
t5 = Task(title = "Read Programming Scala book", dueBy = LocalDateTime.now().plusDays(30), tags = Set("reading", "books", "scala"), priority = Priority.HIGH)
t6 = Task(title = "Go to Goa for holiday", dueBy = LocalDateTime.now().plusDays(60), tags = Set("travel"), priority = Priority.LOW)
t7 = Task(title = "Build my dream application using Play framework and Slick", dueBy = LocalDateTime.now().plusMonths(3), tags = Set("application", "play", "startup"), priority = Priority.HIGH)
val tasks = Seq(t1, t2, t3, t4, t5, t6, t7)
performAction(dataModel.insertTaskAction(tasks: _*))
}
private def performAction[T](action: DBIO[T]): T = {
Await.result(db.run(action), 2 seconds)
}
}
```
In the code shown above, we have done the following:
1. We imported all the required classes and traits that are required by our test case.
2. We provided implementation of `beforeAll` method. This allows us to perform one time setup for this test case. We inserted seven tasks in the database using the `insertTaskAction` we discussed last week. In the task list shown above, there are two tasks that were due in past and 5 tasks which are due in future.
3. `performAction` is a method that will help us avoid writing boilerplate code of wrapping the future in an `Await`. We will just pass an action to `performAction` and it will take care of the rest. We will use this method in all our test cases.
Now, that we have setup our test data. We can write our first test case that will select all the tasks in the `tasks` table.
```scala
import queries._
it("should select all the tasks stored in the database") {
val tasks = performAction(selectAllTasksQuery.result)
tasks should have length 7
tasks.head should have(
'title (t1.title),
'description (t1.description),
'createdAt (t1.createdAt),
'dueBy (t1.dueBy),
'tags (t1.tags)
)
}
```
In the code shown above, only thing that is of interest to us is the `selectAllTasksQuery`. This is imported from the `queries` object. `performAction` method discussed above needs an action. You can convert a query to an action by calling the `result` method on it. If you try to run the test case now, it will not work as we have not yet defined `selectAllTasksQuery`.
In the `queries` object, define `selectAllTasksQuery` as shown below.
```scala
object queries {
val selectAllTasksQuery: Query[TaskTable, Task, Seq] = Tasks
}
```
Let's try to decipher one line of code that we have written above. In the code shown above, we have a defined a value `selectAllTasksQuery` that returns `Tasks` object. `Tasks` is an instance of `TableQuery` object we defined in `dataModels.scala`. `Tasks` i.e. `TableQuery` object is the gateway to the Slick query DSL API. When you return `Tasks` object then Slick uses the default `*` projection that we defined in the `TaskTable`.
The other interesting bit is the type of `selectAllTasksQuery`. You are not required to define the type here as Scala can infer the type. By understanding the type `Query[TaskTable, Task, Seq]`, you will understand how Slick determine what value should be returned by the query. `Query` takes three type parameters. The first type parameter is called the packed type i.e. the type of values you work against in the query DSL. The second type is called the unpacked type i.e. the type of values you get back when you run the query. The third type is the container type that collects the result.
Run the test case and it should pass. You can look at the logs to confirm that Slick executed `select *` query.
```sql
select "title", "description", "createdAt", "dueBy", "tags", "priority", "id" from "tasks"
```
## Select all task titles
The first query that we saw above fetches all the columns of `tasks` table. Most of the time we only want to select few columns. Let's write our test case for this use case.
```scala
it("should select all task titles") {
val taskTitles = performAction(selectAllTaskTitleQuery.result)
taskTitles should have length 7
taskTitles should be(List(t1.title, t2.title, t3.title, t4.title, t5.title, t6.title, t7.title))
}
```
As you can see above, we are executing `selectAllTaskTitleQuery`. This query is defined in `queries` object as shown below.
```scala
val selectAllTaskTitleQuery: Query[Rep[String], String, Seq] = Tasks.map(taskTable => taskTable.title)
```
In the code shown above, we have used map function on the `Tasks` table query object. `map` is a transformation function that take a lambda. The lambda function tells Slick that we only want to select title column. One thing to note here is that in the `map` function we are working on the `TaskTable` object. As `map` function only returns title so the type of `selectAllTaskTitleQuery` is `Query[Rep[String], String, Seq]`.
You can also use the shorthand `_` in the lambda as shown below.
```scala
val selectAllTaskTitleQuery: Query[Rep[String], String, Seq] = Tasks.map(_.title)
```
You can also select more than one columns in the map function as shown below.
```scala
val selectMultipleColumnsQuery: Query[(Rep[String], Rep[Priority], Rep[LocalDateTime]), (String, Priority, LocalDateTime), Seq] = Tasks.map(t => (t.title, t.priority, t.createdAt))
```
The query executed by Slick can be seen in the logs.
```sql
select "title", "priority", "createdAt" from "tasks"
```
## Select all the high priority task titles
So far we have selected all the data in our tasks table. There are times when we have to filter data as we have to do it this usecase. We have filter out all the high priority tasks and then select only title field. Let's write the test case first.
```scala
it("should select all the high priority task titles"){
val highPriorityTasks = performAction(selectHighPriorityTasksQuery.result)
highPriorityTasks should have length 4
highPriorityTasks should be(List(t1.title, t3.title, t5.title, t7.title))
}
```
In the dataset that we created in `beforeAll` method, we have four high priority tasks.
The `selectHighPriorityTasksQuery` will use the `filter` and the `map` operation to get the job done. `filter` allows us to specify the `where` clauses.
```scala
val selectHighPriorityTasksQuery: Query[Rep[String], String, Seq] = Tasks.filter(_.priority === Priority.HIGH).map(_.title)
```
In the code shown above, we first filtered out all the high priority tasks and then selected only title column.
You can view the SQL query generated by Slick in the logs.
```sql
select "title" from "tasks" where "priority" = 3
```
## Paginate results
Slick allows you to paginate our the result by using the `drop` and `limit` methods of `TableQuery`. To skip first 3 elements and then limit the result to 2 records, you can write following Slick code.
```scala
Tasks.drop(3).take(2)
```
You can view the SQL query generated by Slick in the logs.
```sql
select "title", "description", "createdAt", "dueBy", "tags", "priority", "id" from "tasks" limit 3 offset 2
```
## Sort tasks in descending order of due date
A lot of times we have to work with data in some sorting order. Let's suppose, we want to work on the task that is due last. One way to sort would be to sort the data in your application code. You could also ask your database to return the data in sorted order by passing the `order by clause`. Let's write a test case to test this scenario.
```scala
it("should sort tasks in descending order of due date") {
val tasks = performAction(selectTasksSortedByDueDateDescQuery.result)
tasks.head should have(
'title (t7.title),
'description (t7.description),
'createdAt (t7.createdAt),
'dueBy (t7.dueBy),
'tags (t7.tags)
)
}
```
We have to define `selectTasksSortedByDueDateDescQuery` in the `queries` object as shown below.
```scala
val selectTasksSortedByDueDateDescQuery = Tasks.sortBy(_.dueBy.desc)
```
The reason `desc` is available on the `dueBy` is because for Slick it is a `Timestamp`. All the operations that work on `Timestamp` are available on the `dueBy` as well.
You can view the SQL query generated by Slick in the logs.
```sql
select "title", "description", "createdAt", "dueBy", "tags", "priority", "id" from "tasks" order by "dueBy" desc
```
## Select all tasks due today
To select all the tasks due today we can use `filter` operator as shown below. We are using `LocalDate` `asStartOfDay` method to define the time range of our where clause.
```scala
val selectAllTasksDueToday = Tasks
.filter(t => t.dueBy > LocalDate.now().atStartOfDay() && t.dueBy < LocalDate.now().atStartOfDay().plusDays(1))
.map(_.title)
```
You could have also used two filters instead of one as shown below.
```scala
val selectAllTasksDueToday = Tasks
.filter(_.dueBy > LocalDate.now().atStartOfDay())
.filter(_.dueBy < LocalDate.now().atStartOfDay().plusDays(1))
.map(_.title)
```
You can view the SQL query generated by Slick in the logs.
```sql
select "title" from "tasks" where ("dueBy" > {ts '2016-01-31 00:00:00.0'}) and ("dueBy" < {ts '2016-02-01 00:00:00.0'})
```
## Select data with in a range
We can use the SQL `BETWEEN` operator to select data between two dates as shown below.
```scala
val selectTasksBetweenTodayAndSameDateNextMonthQuery = Tasks.filter(t => t.dueBy.between(LocalDateTime.now(), LocalDateTime.now().plusMonths(1)))
```
You can view the SQL query generated by Slick in the logs.
```sql
select "title", "description", "createdAt", "dueBy", "tags", "priority", "id" from "tasks" where "dueBy" between {ts '2016-01-31 21:44:40.643'} and {ts '2016-02-29 21:44:40.643'}
```
## Check if any high priority task is pending today
You can use SQL `exists` operator as shown below.
```scala
val selectAllTasksDueToday = Tasks
.filter(_.dueBy > LocalDate.now().atStartOfDay())
.filter(_.dueBy < LocalDate.now().atStartOfDay().plusDays(1))
val checkIfAnyHighPriorityTaskExistsToday = selectAllTasksDueToday.filter(_.priority === Priority.HIGH).exists
```
You can view the SQL query generated by Slick in the logs.
```sql
select exists(select "description", "createdAt", "priority", "tags", "dueBy", "id", "title" from "tasks" where (("dueBy" > {ts '2016-01-31 00:00:00.0'}) and ("dueBy" < {ts '2016-02-01 00:00:00.0'})) and ("priority" = 3))
```
There are many more aggregate functions like `max`, `min`, `average` that you can use.
## Conclusion
Today, we looked at how we can use the Slick library to query our data. If you have used Scala collections or Java 8 Streams you should feel home. We still haven't covered many other important Slick topics like joins, profiles, working with real databases like MySQL or PostgreSQL, etc. I will write at least one more post about Slick so that we have good understanding of it.
That's all for this week. Please provide your valuable feedback by adding a comment to [https://github.com/shekhargulati/52-technologies-in-2016/issues/7](https://github.com/shekhargulati/52-technologies-in-2016/issues/7).
[](https://github.com/igrigorik/ga-beacon)
================================================
FILE: 05-slick/tasky/.gitignore
================================================
# Created by .ignore support plugin (hsz.mobi)
### SBT template
# Simple Build Tool
# http://www.scala-sbt.org/release/docs/Getting-Started/Directories.html#configuring-version-control
target/
lib_managed/
src_managed/
project/boot/
.history
.cache
### JetBrains template
# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio
*.iml
## Directory-based project format:
.idea/
# if you remove the above rule, at least ignore the following:
# User-specific stuff:
# .idea/workspace.xml
# .idea/tasks.xml
# .idea/dictionaries
# Sensitive or high-churn files:
# .idea/dataSources.ids
# .idea/dataSources.xml
# .idea/sqlDataSources.xml
# .idea/dynamic.xml
# .idea/uiDesigner.xml
# Gradle:
# .idea/gradle.xml
# .idea/libraries
# Mongo Explorer plugin:
# .idea/mongoSettings.xml
## File-based project format:
*.ipr
*.iws
## Plugin-specific files:
# IntelliJ
/out/
# mpeltonen/sbt-idea plugin
.idea_modules/
# JIRA plugin
atlassian-ide-plugin.xml
# Crashlytics plugin (for Android Studio and IntelliJ)
com_crashlytics_export_strings.xml
crashlytics.properties
crashlytics-build.properties
### Scala template
*.class
*.log
# sbt specific
.cache
.history
.lib/
dist/*
target/
lib_managed/
src_managed/
project/boot/
project/plugins/project/
# Scala-IDE specific
.scala_dependencies
.worksheet
================================================
FILE: 05-slick/tasky/build.sbt
================================================
name := "tasky"
description := "A simple task manager for humans"
version := "0.1.0"
scalaVersion := "2.11.7"
libraryDependencies += "com.typesafe.slick" %% "slick" % "3.1.1"
libraryDependencies += "com.h2database" % "h2" % "1.4.191"
libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.1.3"
libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6" % "test"
================================================
FILE: 05-slick/tasky/project/plugins.sbt
================================================
addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.8.1")
================================================
FILE: 05-slick/tasky/src/main/scala/datamodel/Priority.scala
================================================
package datamodel
object Priority extends Enumeration {
type Priority = Value
val HIGH = Value(3)
val MEDIUM = Value(2)
val LOW = Value(1)
}
================================================
FILE: 05-slick/tasky/src/main/scala/datamodel/columnDataMappers.scala
================================================
package datamodel
import java.sql.Timestamp
import java.time.LocalDateTime
import datamodel.Priority._
import slick.driver.H2Driver.api._
object columnDataMappers {
implicit val localDateTimeColumnType: BaseColumnType[LocalDateTime] = MappedColumnType.base[LocalDateTime, Timestamp](
ldt => Timestamp.valueOf(ldt),
t => t.toLocalDateTime
)
implicit val setStringColumnType: BaseColumnType[Set[String]] = MappedColumnType.base[Set[String], String](
tags => tags.mkString(","),
tagsString => tagsString.split(",").toSet
)
implicit val priorityMapper = MappedColumnType.base[Priority, Int](
p => p.id,
v => Priority(v)
)
}
================================================
FILE: 05-slick/tasky/src/main/scala/datamodel/dataModel.scala
================================================
package datamodel
import java.time.LocalDateTime
import datamodel.Priority.Priority
import datamodel.columnDataMappers._
import slick.driver.H2Driver.api._
object dataModel {
case class Task(
title: String,
description: String = "",
createdAt: LocalDateTime = LocalDateTime.now(),
dueBy: LocalDateTime,
tags: Set[String] = Set[String](),
priority: Priority = Priority.LOW,
id: Long = 0L)
class TaskTable(tag: Tag) extends Table[Task](tag, "tasks") {
def title = column[String]("title")
def description = column[String]("description")
def createdAt = column[LocalDateTime]("createdAt")
def dueBy = column[LocalDateTime]("dueBy")
def tags = column[Set[String]]("tags")
def priority = column[Priority]("priority")
def id = column[Long]("id", O.PrimaryKey, O.AutoInc)
override def * = (title, description, createdAt, dueBy, tags, priority, id) <>(Task.tupled, Task.unapply)
}
lazy val Tasks = TableQuery[TaskTable]
val createTaskTableAction = Tasks.schema.create
def insertTaskAction(tasks: Task*) = Tasks ++= tasks.toSeq
val listAllTasksAction = Tasks.result
}
================================================
FILE: 05-slick/tasky/src/main/scala/queries/queries.scala
================================================
package queries
import java.time.{LocalDate, LocalDateTime}
import datamodel.Priority
import datamodel.Priority.Priority
import datamodel.columnDataMappers._
import datamodel.dataModel._
import slick.driver.H2Driver.api._
object queries {
val selectAllTasksQuery: Query[TaskTable, Task, Seq] = Tasks
// val findAllTaskTitleQuery = Tasks.map(taskTable => taskTable.title)
val selectAllTaskTitleQuery: Query[Rep[String], String, Seq] = Tasks.map(_.title)
val selectMultipleColumnsQuery: Query[(Rep[String], Rep[Priority], Rep[LocalDateTime]), (String, Priority, LocalDateTime), Seq] = Tasks.map(t => (t.title, t.priority, t.createdAt))
val selectHighPriorityTasksQuery: Query[Rep[String], String, Seq] = Tasks.filter(_.priority === Priority.HIGH).map(_.title)
def findAllTasksPageQuery(skip: Int, limit: Int) = Tasks.drop(skip).take(limit)
val selectTasksSortedByDueDateDescQuery = Tasks.sortBy(_.dueBy.desc)
val findAllDueTasks = Tasks.filter(_.dueBy >= LocalDate.now().atStartOfDay())
val selectAllTaskTitlesDueToday = Tasks
.filter(_.dueBy > LocalDate.now().atStartOfDay())
.filter(_.dueBy < LocalDate.now().atStartOfDay().plusDays(1))
.map(_.title)
val selectTasksBetweenTodayAndSameDateNextMonthQuery = Tasks.filter(t => t.dueBy.between(LocalDateTime.now(), LocalDateTime.now().plusMonths(1)))
val selectAllTasksDueToday = Tasks
.filter(_.dueBy > LocalDate.now().atStartOfDay())
.filter(_.dueBy < LocalDate.now().atStartOfDay().plusDays(1))
val checkIfAnyHighPriorityTaskExistsToday = selectAllTasksDueToday.filter(_.priority === Priority.HIGH).exists
}
================================================
FILE: 05-slick/tasky/src/test/resources/application.conf
================================================
taskydb = {
connectionPool = disabled
url = "jdbc:h2:mem:taskydb"
driver = "org.h2.Driver"
keepAliveConnection = true
}
================================================
FILE: 05-slick/tasky/src/test/scala/datamodel/DataModelSpec.scala
================================================
package datamodel
import java.time.LocalDateTime
import datamodel.dataModel.Task
import org.scalatest.{BeforeAndAfterEach, FunSpec, Matchers}
import slick.driver.H2Driver.api._
import scala.concurrent._
import scala.concurrent.duration._
class DataModelSpec extends FunSpec with Matchers with BeforeAndAfterEach {
var db: Database = _
override protected def beforeEach(): Unit = {
db = Database.forConfig("taskydb")
Await.result(db.run(dataModel.createTaskTableAction), 2 seconds)
}
override protected def afterEach(): Unit = db.shutdown
describe("DataModel Spec") {
it("should insert single task into database") {
val result = Await.result(db.run(dataModel.insertTaskAction(Task(title = "Learn Slick", dueBy = LocalDateTime.now().plusDays(1), priority = Priority.HIGH))), 2 seconds)
result should be(Some(1))
}
it("should insert multiple tasks into database") {
val tasks = Seq(
Task(title = "Learn Slick", dueBy = LocalDateTime.now().plusDays(1), priority = Priority.HIGH),
Task(title = "Write blog on Slick", dueBy = LocalDateTime.now().plusDays(2), priority = Priority.HIGH),
Task(title = "Build a simple application using Slick", dueBy = LocalDateTime.now().plusDays(3), priority = Priority.HIGH)
)
val result = Await.result(db.run(dataModel.insertTaskAction(tasks: _*)), 2 seconds)
result should be(Some(3))
}
it("should list all tasks in the database") {
val tasks = Seq(
Task(title = "Learn Slick", dueBy = LocalDateTime.now().plusDays(1), priority = Priority.HIGH),
Task(title = "Write blog on Slick", dueBy = LocalDateTime.now().plusDays(2), priority = Priority.HIGH),
Task(title = "Build a simple application using Slick", dueBy = LocalDateTime.now().plusDays(3), priority = Priority.HIGH)
)
Await.result(db.run(dataModel.insertTaskAction(tasks: _*)), 2 seconds)
val result = Await.result(db.run(dataModel.listAllTasksAction), 2 seconds)
result should have length 3
}
}
}
================================================
FILE: 05-slick/tasky/src/test/scala/queries/QueriesSpec.scala
================================================
package queries
import java.time.LocalDateTime
import datamodel.dataModel.Task
import datamodel.{Priority, dataModel}
import org.scalatest.{BeforeAndAfterAll, FunSpec, Matchers}
import queries._
import slick.driver.H2Driver.api._
import scala.concurrent._
import scala.concurrent.duration._
class QueriesSpec extends FunSpec with Matchers with BeforeAndAfterAll {
var db: Database = _
var t1: Task = _
var t2: Task = _
var t3: Task = _
var t4: Task = _
var t5: Task = _
var t6: Task = _
var t7: Task = _
override protected def beforeAll(): Unit = {
db = Database.forConfig("taskydb")
Await.result(db.run(dataModel.createTaskTableAction), 2 seconds)
t1 = Task(title = "Write part 1 blog on Slick", dueBy = LocalDateTime.now().minusDays(7), tags = Set("blogging", "scala", "slick"), priority = Priority.HIGH)
t2 = Task(title = "Give a Java 8 training", dueBy = LocalDateTime.now().minusDays(3), tags = Set("java", "training", "travel"), priority = Priority.LOW)
t3 = Task(title = "Write part 2 blog on Slick queries", dueBy = LocalDateTime.now(), tags = Set("blogging", "scala", "slick"), priority = Priority.HIGH)
t4 = Task(title = "Read Good to Great book", dueBy = LocalDateTime.now().plusDays(15), tags = Set("reading", "books", "startup"), priority = Priority.MEDIUM)
t5 = Task(title = "Read Programming Scala book", dueBy = LocalDateTime.now().plusDays(30), tags = Set("reading", "books", "scala"), priority = Priority.HIGH)
t6 = Task(title = "Go to Goa for holiday", dueBy = LocalDateTime.now().plusDays(60), tags = Set("travel"), priority = Priority.LOW)
t7 = Task(title = "Build my dream application using Play framework and Slick", dueBy = LocalDateTime.now().plusMonths(3), tags = Set("application", "play", "startup"), priority = Priority.HIGH)
val tasks = Seq(t1, t2, t3, t4, t5, t6, t7)
performAction(dataModel.insertTaskAction(tasks: _*))
}
private def performAction[T](action: DBIO[T]): T = {
Await.result(db.run(action), 2 seconds)
}
describe("Task Data Model Query Spec") {
it("should select all the tasks stored in the database") {
val tasks = performAction(selectAllTasksQuery.result)
tasks should have length 7
tasks.head should have(
'title (t1.title),
'description (t1.description),
'createdAt (t1.createdAt),
'dueBy (t1.dueBy),
'tags (t1.tags)
)
}
it("should select all task titles") {
val taskTitles = performAction(selectAllTaskTitleQuery.result)
taskTitles should have length 7
taskTitles should be(List(t1.title, t2.title, t3.title, t4.title, t5.title, t6.title, t7.title))
}
it("should select task title, priority, and creation date for all tasks") {
val taskTitles = performAction(selectMultipleColumnsQuery.result)
taskTitles should have length 7
taskTitles should be(List(
(t1.title, t1.priority, t1.createdAt),
(t2.title, t2.priority, t2.createdAt),
(t3.title, t3.priority, t3.createdAt),
(t4.title, t4.priority, t4.createdAt),
(t5.title, t5.priority, t5.createdAt),
(t6.title, t6.priority, t6.createdAt),
(t7.title, t7.priority, t7.createdAt))
)
}
it("should select all the high priority task titles"){
val highPriorityTasks = performAction(selectHighPriorityTasksQuery.result)
highPriorityTasks should have length 4
highPriorityTasks should be(List(t1.title, t3.title, t5.title, t7.title))
}
it("should skip first 2 records and then limit result to 3") {
val tasks = performAction(findAllTasksPageQuery(2, 3).result)
tasks should have length 3
tasks.head should have(
'title (t3.title),
'description (t3.description),
'createdAt (t3.createdAt),
'dueBy (t3.dueBy),
'tags (t3.tags)
)
}
it("should sort tasks in descending order of due date") {
val tasks = performAction(selectTasksSortedByDueDateDescQuery.result)
tasks.head should have(
'title (t7.title),
'description (t7.description),
'createdAt (t7.createdAt),
'dueBy (t7.dueBy),
'tags (t7.tags)
)
}
it("should find all due tasks") {
val dueTasks = performAction(findAllDueTasks.result)
dueTasks should have length 5
dueTasks.map(_.title) should be(List(t3.title, t4.title, t5.title, t6.title, t7.title))
}
it("should find all tasks due today") {
val dueTasks = performAction(selectAllTaskTitlesDueToday.result)
dueTasks should have length 1
dueTasks should be(List(t3.title))
}
it("select tasks between today and same date next month "){
val tasks = performAction(selectTasksBetweenTodayAndSameDateNextMonthQuery.result)
tasks should have length 1
}
it("check if any high priority task exists today"){
val exists = performAction(checkIfAnyHighPriorityTaskExistsToday.result)
exists should be(true)
}
}
}
================================================
FILE: 06-okhttp/README.md
================================================
Building A Lightweight Scala REST API Client with OkHttp
----
Welcome to the sixth blog of [52-technologies-in-2016](https://github.com/shekhargulati/52-technologies-in-2016) blog series. In this blog, we will learn how to write Scala REST API client for [Medium](https://medium.com/)'s REST API using [OkHttp](https://github.com/square/okhttp) library. [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) APIs have become a standard method of communication between two devices over a network. Most applications expose their REST API that developers can use to get work with an application programmatically. For example, if I have to build a realtime opinion mining application then I can use Twitter or Facebook REST APIs to get hold of their data and build my application. To work with an application REST APIs, you either can write your own client or you can use one of the language specific client provided by the application. Last few weeks, I have started using [Medium](https://medium.com/) for posting non-technical blogs. Medium is a blog publishing platform created by Twitter co-founder Evan Williams. Evan Williams is the same guy who earlier created Blogger, which was bought by Google in 2003.
Medium exposed their REST API to the external world last year. The API is simple and allows you to do operations like submitting a post, getting details of the authenticated user, getting publications for a user, etc. You can read about Medium API documentation in their [Github repository](https://github.com/Medium/medium-api-docs). Medium officially provides REST API clients for [Node.js](https://github.com/Medium/medium-sdk-nodejs), [Python](https://github.com/Medium/medium-sdk-python), and [Go](https://github.com/Medium/medium-sdk-go) programming languages. I couldn't find Scala client for Medium REST API so I decided to write my own client using OkHttp.
## What is OkHttp?
[OkHttp](https://square.github.io/okhttp/) is an open source Java HTTP client library focussed on efficiency. It is written by folks at [Square](https://squareup.com/). It supports [SPDY](https://developers.google.com/speed/spdy/), [HTTP/2](https://http2.github.io/), and [WebSocket](https://tools.ietf.org/html/rfc6455) protocols.
OkHttp API is very easy to use. You just have to add its dependency to your classpath and then you can start using it to build your clients.
According to [OkHttp documentation](https://square.github.io/okhttp/),
> OkHttp is an HTTP client that’s efficient by default:
* HTTP/2 support allows all requests to the same host to share a socket.
* Connection pooling reduces request latency (if HTTP/2 isn’t available).
* Transparent GZIP shrinks download sizes.
* Response caching avoids the network completely for repeat requests.
## Why are you using a Java library?
I know you must be thinking why I am using a Java library to build a Scala REST API client. Like most Scala developers, I thought of using a Scala library instead. But, as I started looking into which Scala library should I use I didn't find any single winner. If you search for "Scala REST client", you will land up on this [StackOverFlow question](https://stackoverflow.com/questions/12334476/simple-and-concise-http-client-library-for-scala). It suggests four libraries Dispatch, Scalaz http, spray-client, Play WS. Let's discuss why I didn't used them one by one.
1. [Dispatch](https://github.com/dispatch/reboot): It is a Scala wrapper around Ning's Async Http Client. The project doesn't look very active with last commit on [May 30, 2015](https://github.com/dispatch/reboot/commits/0.11.3). The [travis-ci build](https://travis-ci.org/dispatch/reboot) is also broken so I am not sure if this project is actively maintained.
2. [Scalaz](https://github.com/scalaz/scalaz/) http: Scalaz is an extension to the core Scala library for functional programming. They used to have an HTTP client. They [dropped http module from Scalaz in version 7](https://stackoverflow.com/questions/25482520/what-happened-to-the-scalaz-http-module).
3. [spray-client](http://spray.io/documentation/1.2.2/spray-client/): It provides high-level HTTP client functionality by adding another logic layer on top of the relatively basic spray-can HTTP Client APIs. spray-client depends on many other spray projects and Akka. I didn't wanted to use a library that depends on so many other libraries.
4. [Play WS](https://www.playframework.com/documentation/2.5.x/ScalaWS): Play WS is part of the Scala's Play web framework. It can used in standalone mode but it also depends on [many other libraries](http://mvnrepository.com/artifact/com.typesafe.play/play-ws_2.11/2.4.6). It also looked very heavy weight for something simple. So, I decided not to use it as well.
## Why OkHttp?
My reasons for going with OkHttp are:
1. It has only one dependency Okio. [Okio](https://github.com/square/okio) is a library that complements java.io and java.nio to make it much easier to access, store, and process your data.
2. It has very good testing support. It provides [scriptable web server](https://github.com/square/okhttp/tree/master/mockwebserver) for testing HTTP client. This makes it easy to test whether your client is doing the right thing without depending on the network.
3. OkHttp is one of the few libraries that is designed up front for efficiency.
4. Stable and actively developed by Square. Last commit was 15 hours ago.
5. API is very simple and intuitive to use. It comes with good defaults and works like a charm.
Although, OkHttp is a Java library but it works great with Scala. I know it might not be the Scala way but sometimes we have to become pragmatic and choose the right tool for the job. There is also an OkHttp Scala wrapper called [Communicator](https://github.com/Taig/Communicator) that one can use.
## Github repository
The code for today’s application is available on github: [medium-scala-client](./medium-scala-client). In this blog, I will only cover couple of REST endpoints. You can view the full source of [medium-scala-sdk here](https://github.com/shekhargulati/medium-scala-sdk).
## Getting Started
Start by creating a new directory `medium-scala-client` at a convenient location on your filesystem. This directory will house the source code of our client.
```bash
$ mkdir medium-scala-client
```
Create a new file `build.sbt` inside the `medium-scala-client` directory. `build.sbt` is the sbt build script.
> **If you are new to sbt, then [please refer to my earlier post on it](../02-sbt/README.md).**
Populate `build.sbt` with following contents.
```scala
name := "medium-scala-client"
version := "1.0"
description := "Scala client for Medium.com REST API"
scalaVersion := "2.11.7"
libraryDependencies += "com.squareup.okhttp3" % "okhttp" % "3.0.1"
libraryDependencies += "io.spray" %% "spray-json" % "1.3.2"
libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6" % "test"
libraryDependencies += "com.squareup.okhttp3" % "mockwebserver" % "3.0.1" % "test"
```
In the build script shown above, you can see that we have only added two compile time dependencies -- `okhttp` and `spray-json`. [spray-json](https://github.com/spray/spray-json) is a lightweight, clean, and efficient library to work with JSON in Scala. It has no dependencies. We will use it to convert our domain objects into JSON and vice-versa. `scalatest` and `mockwebserver` are added for testing.
Create a project layout for your Scala source and test files.
```bash
$ mkdir -p src/main/{scala,resources}
$ mkdir -p src/test/scala
```
## Getting the authenticated user's details
Let's start with implementing the REST endpoint to get details of an authenticated user. To get the details of a user, we have to make an HTTP GET request.
```
GET https://api.medium.com/v1/me
```
We will start with writing a test. Create a new package `medium` inside the `src/test/scala`. After creating the package, create a Scala class `MediumClientSpec`. Populate the `MediumClientSpec` with following contents.
```scala
package medium
import okhttp3.mockwebserver.MockWebServer
import org.scalatest.{BeforeAndAfterEach, FunSpec, Matchers}
class MediumClientSpec extends FunSpec with Matchers with BeforeAndAfterEach{
var server: MockWebServer = _
override protected def beforeEach(): Unit = {
server = new MockWebServer()
}
override protected def afterEach(): Unit = {
server.shutdown()
}
}
```
The code shown above does the following:
1. We created a new class `MediumClientSpec` that extended `FunSpec`, `Matchers`, and `BeforeAndAfterEach` traits. These are part of `scalatest` library.
2. We override two methods of `BeforeAndAfterEach` trait. `beforeEach` will make sure that `MockWebServer` instance is created before each test case is executed. `MockWebServer` is a scriptable web server. You can configure it to return mock responses for your requests. It works very similarly to any mocking framework. You first set your expectations, then run the application code, and finally verify that expected requests were made.
3. `afterEach` will make sure that server is shutdown after each test.
Add the following test case to the `MediumClientSpec`. This code should be added after the `afterEach` method.
```scala
describe("MediumClientSpec") {
it("should get details of an authenticated user") {
val json =
"""
|{
| "data": {
| "id": "123",
| "username": "shekhargulati",
| "name": "Shekhar Gulati",
| "url": "https://medium.com/@shekhargulati",
| "imageUrl": "https://cdn-images-1.medium.com/fit/c/200/200/1*pC-eYQUV-iP2Y10_LgGvwA.jpeg"
| }
|}
""".stripMargin
server.enqueue(new MockResponse()
.setBody(json)
.setHeader("Content-Type", "application/json")
.setHeader("charset", "utf-8"))
server.start()
val medium = new MediumClient("test_client_id", "test_client_secret", Some("access_token")) {
override val baseApiUrl = server.url("/v1/me")
}
val user = medium.getUser
user should have(
'id ("123"),
'username ("shekhargulati"),
'name ("Shekhar Gulati"),
'url ("https://medium.com/@shekhargulati"),
'imageUrl ("https://cdn-images-1.medium.com/fit/c/200/200/1*pC-eYQUV-iP2Y10_LgGvwA.jpeg")
)
}
}
```
Let's understand the code show above:
1. We created a json that will be returned by `MockWebServer` when GET request is made to `https://api.medium.com/v1/me`.
2. Then, we set up the server with a mock response. We set the body to the json created in step 1. Also, we added HTTP headers that will be passed in the response.
3. Next, we started the mock web server so that it can accept test requests.
4. Then, we created an instance of MediumClient(that we will create later in the blog). We have to set the URL returned by our server in the client so that it makes requests to the mock server instead of hitting the actual Medium API. This is the reason we have overridden `baseApiUrl` value of `MediumClient`.
5. Finally, we called the `getUser` method of `MediumClient` and asserted its response.
Now that we have written our test case we should start working on the implementation of MediumClient. Create a new package `medium` inside `src/main/scala`. Then, create a new Scala class `MediumClient` inside the `medium` package.
```scala
package medium
class MediumClient(clientId: String, clientSecret: String, var accessToken: Option[String] = None)
object MediumClient {
def apply(clientId: String, clientSecret: String): MediumClient = new MediumClient(clientId, clientSecret)
def apply(clientId: String, clientSecret: String, accessToken: String): MediumClient = new MediumClient(clientId, clientSecret, Some(accessToken))
}
case class MediumException(message: String, cause: Throwable = null) extends RuntimeException(message, cause)
```
The code shown above does the following:
1. We created a new Scala class `MediumClient`. The primary constructor of MediumClient takes three arguments -- `clientId`, `clientSecret`, and `accessToken`. The clientId and clientSecret are created for you when you create new Medium application [http://medium.com/me/applications](http://medium.com/me/applications). Using the clientId and clientSecret, users can generate `accessToken`. You have to pass `accessToken` in each request to the Medium API.
2. Then, we created a companion object to the `MediumClient`. It provides factory methods to easily construct `MediumClient` instances.
3. `MediumException` is a runtime exception that we will throw when client will not be able to process user requests.
Create an instance of `OkHttpClient` inside the `MediumClient` class as shown below. `OkHttpClient` is used to send HTTP requests and read HTTP responses. When you create the `OkHttpClient` instance using the default constructor then an `OkHttpClient` instance is created using the default values. You can also create an instance configured using other values by using the `OkHttpClient.Builder` API. We also created another value `baseApiUrl` of type `okhttp3.HttpUrl`. This will store the base URL of the Medium API i.e. `https://api.medium.com`.
```scala
import okhttp3.{HttpUrl, OkHttpClient}
class MediumClient(clientId: String, clientSecret: String, var accessToken: Option[String] = None) {
val client = new OkHttpClient()
val baseApiUrl: HttpUrl = new HttpUrl.Builder()
.scheme("https")
.host("api.medium.com")
.build()
}
```
Now, we will write the `getUser` method that will make an HTTP GET request to the Medium API to fetch the user details. User is determined using the `accessToken`. If access token is not set then it will throw `MediumException`.
```scala
def getUser: User = accessToken match {
case Some(at) => ???
case _ => throw new MediumException("Please set access token")
}
```
> The Scala syntax `???` lets you write a not yet implemented method. This allows you to write code that compiles. But, if you run this code, then it will thrown an exception.
The code shown above needs `User` to compile. Create a new Scala object `domainObjects`. The `domainObjects.scala` will house all our domain objects like `User`, `Post`, etc. Create a case class for `User` inside it as shown below.
```scala
package medium
object domainObjects {
case class User(id: String, username: String, name: String, url: String, imageUrl: String)
}
```
As shown above, we created a User case class with five fields inside the `domainObjects` Scala object.
After creating the `domainObjects` Scala object, add its import in the `MediumClient` so that code can compile.
```scala
import domainObjects._
class MediumClient(clientId: String, clientSecret: String, var accessToken: Option[String] = None)
```
Now, let's write code to replace `???` with actual implementation inside the `getUser` method.
```scala
def getUser: User = accessToken match {
case Some(at) =>
val request = new Request.Builder()
.header("Content-Type", "application/json")
.header("Accept", "application/json")
.header("Accept-Charset", "utf-8")
.header("Authorization", s"Bearer $at")
.url(baseApiUrl.resolve("/v1/me"))
.get()
.build()
makeRequest[User](request)
case _ => throw new MediumException("Please set access token")
}
private def makeRequest[T](request: Request)(implicit p: JsonReader[T]): T = ???
```
In the code shown above:
1. We created request using the OkHttp `Request.Builder` API. We set the required headers in the request and set url of the request to `/v1/me`. `HttpUrl.resolve` method resolves the url against the baseApiUrl. So, the full url will become `https://api.medium.com/v1/me`. OkHttp understands which HTTP method to use by looking at the request. As you can see above, we called the `get` method of the request builder. This constructs an immutable `okhttp3.Request` object.
2. Once request is created, we passed the request to `makeRequest` method. This method will process any request be it GET or POST or DELETE and return the domain object.
Now, we will implement `makeRequest` method. `makeRequest` method makes use of `OkHttpClient` instance to create a new `Call`. To make the HTTP call, we first called `newCall` method on `OkHttpClient` instance. The `newCall` returns `okhttp3.Call` object. OkHttp uses `Call` to model the task of satisfying your request through however many intermediate requests and responses are necessary. Calls can be executed in synchronous or asynchronous manner. In the code shown below, we called the `execute` method to make a synchronous HTTP GET call.
```scala
private def makeRequest[T](request: Request)(implicit p: JsonReader[T]): T= {
val response = client.newCall(request).execute()
val responseJson = response.body().string()
println(s"Received response $responseJson")
???
}
```
If you run the test method now, it will render the `json` response we have set in the test.
```javascript
Received response
{
"data": {
"id": "123",
"username": "shekhargulati",
"name": "Shekhar Gulati",
"url": "https://medium.com/@shekhargulati",
"imageUrl": "https://cdn-images-1.medium.com/fit/c/200/200/1*pC-eYQUV-iP2Y10_LgGvwA.jpeg"
}
}
```
Now, let's take a look at the last bit of code required to convert json into `User` object. To convert json into User object, we will make use of `spray-json` library.
To use `spray-json`, we have to first add few imports so that relevant elements are added in the scope of our `MediumClient`.
```scala
import spray.json._
import DefaultJsonProtocol._
```
After adding the imports, you can convert the json string into User object as shown below.
```scala
private def makeRequest[T](request: Request)(implicit p: JsonReader[T]): T= {
val response = client.newCall(request).execute()
val responseJson = response.body().string()
println(s"Received response $responseJson")
response match {
case r if r.isSuccessful =>
val jsValue: JsValue = responseJson.parseJson
jsValue.asJsObject.getFields("data").headOption match {
case Some(data) => data.convertTo[T]
case _ => throw new MediumException(s"Received unexpected JSON response $responseJson")
}
case _ => throw new MediumException(s"Received HTTP error response code ${response.code()}")
}
}
```
The code shown above will not compile as you have to bring implicit values in scope that provide `JsonFormat[User]` instances for User.
Create a new object `MediumApiProtocol` that will define a `JsonFormat` to convert `User` into JSON.
```scala
package medium
import medium.domainObjects.User
import spray.json.DefaultJsonProtocol
object MediumApiProtocol extends DefaultJsonProtocol{
implicit val userFormat = jsonFormat5(User)
}
```
Now, code will compile and test case will pass.
## Posting a blog on Medium
Let's now implement method that will create a post on Medium. To create a post, we have to use HTTP POST method as we are creating a resource on the server. Let's write a test method, that will test the post creation.
```scala
it("should publish a new post") {
val responsJson =
"""
|{
| "data": {
| "id": "e6f36a",
| "title": "Liverpool FC",
| "authorId": "5303d74c64f66366f00cb9b2a94f3251bf5",
| "tags": ["football", "sport", "Liverpool"],
| "url": "https://medium.com/@majelbstoat/liverpool-fc-e6f36a",
| "canonicalUrl": "http://jamietalbot.com/posts/liverpool-fc",
| "publishStatus": "public",
| "publishedAt": 1442286338435,
| "license": "all-rights-reserved",
| "licenseUrl": "https://medium.com/policy/9db0094a1e0f"
| }
|}
""".stripMargin
server.enqueue(new MockResponse()
.setBody(responsJson)
.setHeader("Content-Type", "application/json")
.setHeader("charset", "utf-8"))
server.start()
val medium = new MediumClient("test_client_id", "test_client_secret", Some("access_token")) {
override val baseApiUrl = server.url("/v1/users/123/posts")
}
val content =
"""
|# Hello World
|Hello how are you?
|## What's up today?
|Writing REST client for Medium API
""".stripMargin
val post = medium.createPost("123", PostRequest("Liverpool FC", "html", content))
post.id should be("e6f36a")
}
```
Next, add `PostRequest` and `Post` case classes to `domainObjects.scala`.
```scala
package medium
object domainObjects {
case class User(id: String, username: String, name: String, url: String, imageUrl: String)
case class PostRequest(title: String, contentFormat: String, content: String, tags: Array[String] = Array(), canonicalUrl: Option[String] = None, publishStatus: String = "public", license: String = "all-rights-reserved")
case class Post(id: String, publicationId: Option[String] = None, title: String, authorId: String, tags: Array[String], url: String, canonicalUrl: String, publishStatus: String, publishedAt: Long, license: String, licenseUrl: String)
}
```
Write the JSON formatter in `MediumApiProtocol` as shown below.
```scala
package medium
import medium.domainObjects._
import spray.json._
object MediumApiProtocol extends DefaultJsonProtocol{
implicit val userFormat = jsonFormat5(User)
implicit val postRequestFormat = jsonFormat7(PostRequest)
implicit val postFormat = jsonFormat11(Post)
}
```
Now, we will write `createPost` that will create a Medium post.
```scala
def createPost(authorId: String, postRequest: PostRequest): Post = accessToken match {
case Some(at) =>
val httpUrl = baseApiUrl.resolve(s"/v1/users/$authorId/posts")
val request = new Request.Builder()
.header("Content-Type", "application/json")
.header("Accept", "application/json")
.header("Accept-Charset", "utf-8")
.header("Authorization", s"Bearer $at")
.url(httpUrl)
.post(RequestBody.create(MediaType.parse("application/json"), postRequest.toJson.prettyPrint))
.build()
makeRequest[Post](request)
case _ => throw new MediumException("Please set access token")
}
```
Now, compile the code and run the test case. Both test cases will pass now.
## Conclusion
This week we learnt how to write REST API using OkHttp library. We covered how to make HTTP GET and POST requests using OkHttp. OkHttp supports all HTTP methods like head, delete, put, etc. You can also use OKHttp to [make asynchronous calls](https://github.com/square/okhttp/wiki/Recipes#asynchronous-get). You can refer to [OkHttp documentation](https://github.com/square/okhttp/wiki) for more details.
That's all for this week. Please provide your valuable feedback by adding a comment to [https://github.com/shekhargulati/52-technologies-in-2016/issues/8](https://github.com/shekhargulati/52-technologies-in-2016/issues/8).
[](https://github.com/igrigorik/ga-beacon)
================================================
FILE: 06-okhttp/medium-scala-client/.gitignore
================================================
# Created by .ignore support plugin (hsz.mobi)
### JetBrains template
# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio
*.iml
## Directory-based project format:
.idea/
# if you remove the above rule, at least ignore the following:
# User-specific stuff:
# .idea/workspace.xml
# .idea/tasks.xml
# .idea/dictionaries
# Sensitive or high-churn files:
# .idea/dataSources.ids
# .idea/dataSources.xml
# .idea/sqlDataSources.xml
# .idea/dynamic.xml
# .idea/uiDesigner.xml
# Gradle:
# .idea/gradle.xml
# .idea/libraries
# Mongo Explorer plugin:
# .idea/mongoSettings.xml
## File-based project format:
*.ipr
*.iws
## Plugin-specific files:
# IntelliJ
/out/
# mpeltonen/sbt-idea plugin
.idea_modules/
# JIRA plugin
atlassian-ide-plugin.xml
# Crashlytics plugin (for Android Studio and IntelliJ)
com_crashlytics_export_strings.xml
crashlytics.properties
crashlytics-build.properties
### SBT template
# Simple Build Tool
# http://www.scala-sbt.org/release/docs/Getting-Started/Directories.html#configuring-version-control
target/
lib_managed/
src_managed/
project/boot/
.history
.cache
### Scala template
*.class
*.log
# sbt specific
.cache
.history
.lib/
dist/*
target/
lib_managed/
src_managed/
project/boot/
project/plugins/project/
# Scala-IDE specific
.scala_dependencies
.worksheet
================================================
FILE: 06-okhttp/medium-scala-client/build.sbt
================================================
name := "medium-scala-client"
version := "1.0"
description := "Scala client for Medium.com REST API"
scalaVersion := "2.11.7"
libraryDependencies += "com.squareup.okhttp3" % "okhttp" % "3.0.1"
libraryDependencies += "io.spray" %% "spray-json" % "1.3.2"
libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6" % "test"
libraryDependencies += "com.squareup.okhttp3" % "mockwebserver" % "3.0.1" % "test"
================================================
FILE: 06-okhttp/medium-scala-client/src/main/scala/medium/MediumApiProtocol.scala
================================================
package medium
import medium.domainObjects._
import spray.json._
object MediumApiProtocol extends DefaultJsonProtocol{
implicit val userFormat = jsonFormat5(User)
implicit val postRequestFormat = jsonFormat7(PostRequest)
implicit val postFormat = jsonFormat11(Post)
}
================================================
FILE: 06-okhttp/medium-scala-client/src/main/scala/medium/MediumClient.scala
================================================
package medium
import medium.MediumApiProtocol._
import medium.domainObjects._
import okhttp3._
import spray.json._
class MediumClient(clientId: String, clientSecret: String, var accessToken: Option[String] = None) {
val client = new OkHttpClient()
val baseApiUrl: HttpUrl = new HttpUrl.Builder()
.scheme("https")
.host("api.medium.com")
.build()
def getUser: User = accessToken match {
case Some(at) =>
val request = new Request.Builder()
.header("Content-Type", "application/json")
.header("Accept", "application/json")
.header("Accept-Charset", "utf-8")
.header("Authorization", s"Bearer $at")
.url(baseApiUrl.resolve("/v1/me"))
.get()
.build()
makeRequest[User](request)
case _ => throw new MediumException("Please set access token")
}
def createPost(authorId: String, postRequest: PostRequest): Post = accessToken match {
case Some(at) =>
val httpUrl = baseApiUrl.resolve(s"/v1/users/$authorId/posts")
val request = new Request.Builder()
.header("Content-Type", "application/json")
.header("Accept", "application/json")
.header("Accept-Charset", "utf-8")
.header("Authorization", s"Bearer $at")
.url(httpUrl)
.post(RequestBody.create(MediaType.parse("application/json"), postRequest.toJson.prettyPrint))
.build()
makeRequest[Post](request)
case _ => throw new MediumException("Please set access token")
}
private def makeRequest[T](request: Request)(implicit p: JsonReader[T]): T= {
val response = client.newCall(request).execute()
val responseJson = response.body().string()
println(s"Received response $responseJson")
response match {
case r if r.isSuccessful =>
val jsValue: JsValue = responseJson.parseJson
jsValue.asJsObject.getFields("data").headOption match {
case Some(data) => data.convertTo[T]
case _ => throw new MediumException(s"Received unexpected JSON response $responseJson")
}
case _ => throw new MediumException(s"Received HTTP error response code ${response.code()}")
}
}
}
object MediumClient {
def apply(clientId: String, clientSecret: String): MediumClient = new MediumClient(clientId, clientSecret)
def apply(clientId: String, clientSecret: String, accessToken: String): MediumClient = new MediumClient(clientId, clientSecret, Some(accessToken))
}
case class MediumException(message: String, cause: Throwable = null) extends RuntimeException(message, cause)
================================================
FILE: 06-okhttp/medium-scala-client/src/main/scala/medium/domainObjects.scala
================================================
package medium
object domainObjects {
case class User(id: String, username: String, name: String, url: String, imageUrl: String)
case class PostRequest(title: String, contentFormat: String, content: String, tags: Array[String] = Array(), canonicalUrl: Option[String] = None, publishStatus: String = "public", license: String = "all-rights-reserved")
case class Post(id: String, publicationId: Option[String] = None, title: String, authorId: String, tags: Array[String], url: String, canonicalUrl: String, publishStatus: String, publishedAt: Long, license: String, licenseUrl: String)
}
================================================
FILE: 06-okhttp/medium-scala-client/src/test/scala/medium/MediumClientSpec.scala
================================================
package medium
import medium.domainObjects.PostRequest
import okhttp3.mockwebserver.{MockResponse, MockWebServer}
import org.scalatest.{BeforeAndAfterEach, FunSpec, Matchers}
class MediumClientSpec extends FunSpec with Matchers with BeforeAndAfterEach {
var server: MockWebServer = _
override protected def beforeEach(): Unit = {
server = new MockWebServer()
}
override protected def afterEach(): Unit = {
server.shutdown()
}
describe("MediumClientSpec") {
it("should get details of an authenticated user") {
val json =
"""
|{
| "data": {
| "id": "123",
| "username": "shekhargulati",
| "name": "Shekhar Gulati",
| "url": "https://medium.com/@shekhargulati",
| "imageUrl": "https://cdn-images-1.medium.com/fit/c/200/200/1*pC-eYQUV-iP2Y10_LgGvwA.jpeg"
| }
|}
""".stripMargin
server.enqueue(new MockResponse()
.setBody(json)
.setHeader("Content-Type", "application/json")
.setHeader("charset", "utf-8"))
server.start()
val medium = new MediumClient("test_client_id", "test_client_secret", Some("access_token")) {
override val baseApiUrl = server.url("/v1/me")
}
val user = medium.getUser
user should have(
'id ("123"),
'username ("shekhargulati"),
'name ("Shekhar Gulati"),
'url ("https://medium.com/@shekhargulati"),
'imageUrl ("https://cdn-images-1.medium.com/fit/c/200/200/1*pC-eYQUV-iP2Y10_LgGvwA.jpeg")
)
}
it("should publish a new post") {
val responsJson =
"""
|{
| "data": {
| "id": "e6f36a",
| "title": "Liverpool FC",
| "authorId": "5303d74c64f66366f00cb9b2a94f3251bf5",
| "tags": ["football", "sport", "Liverpool"],
| "url": "https://medium.com/@majelbstoat/liverpool-fc-e6f36a",
| "canonicalUrl": "http://jamietalbot.com/posts/liverpool-fc",
| "publishStatus": "public",
| "publishedAt": 1442286338435,
| "license": "all-rights-reserved",
| "licenseUrl": "https://medium.com/policy/9db0094a1e0f"
| }
|}
""".stripMargin
server.enqueue(new MockResponse()
.setBody(responsJson)
.setHeader("Content-Type", "application/json")
.setHeader("charset", "utf-8"))
server.start()
val medium = new MediumClient("test_client_id", "test_client_secret", Some("access_token")) {
override val baseApiUrl = server.url("/v1/users/123/posts")
}
val content =
"""
|# Hello World
|Hello how are you?
|## What's up today?
|Writing REST client for Medium API
""".stripMargin
val post = medium.createPost("123", PostRequest("Liverpool FC", "html", content))
post.id should be("e6f36a")
}
}
}
================================================
FILE: 07-hugo/README.md
================================================
Hugo: A Modern WebSite Engine That Just Works
----
This week I decided to take a break from Scala and scratch my own itch my building an online bookshelf using Hugo. **[Hugo](https://gohugo.io/)** is a static site generator written in Go programming language. You can use it for building modern static websites. Static site generator takes your content files written in a markup language like [Markdown](https://en.wikipedia.org/wiki/Markdown), apply layouts you have defined, and generate static HTML files that can be delivered to the user. Static websites are nothing new, they date back to the [first ever website](http://info.cern.ch/hypertext/WWW/TheProject.html) in human history. We started with static websites, then moved to dynamic websites, and finally we are moving back to static websites for use-cases where it make sense. Most common use-cases for static websites are blogs, product documentation, help guides, tutorials, online portfolio or resume.
> **This blog is part of my year long blog series [52 Technologies in 2016](https://github.com/shekhargulati/52-technologies-in-2016)**
Static generators again came into limelight after the introduction of [Jekyll](https://jekyllrb.com/) in 2008. Jekyll is a static website generator written in Ruby. It was created by Github co-founder Tom Preston-Werner. Because Jekyll was created by Github co-founder, it has very good integration with Github. It was very easy to get your website running on Github pages.
Another reason static site generators are back in popularity has to do with a lot of advantages they offer. In my opinion, static generators offer following advantages:
1. You don't need a database to store content
2. Forces you to use version control system to store content
3. Fast and cacheable
4. Less maintenance overhead
5. Works well for a lot of use-cases like blogs, documentation, etc.
6. Low barrier to entry
7. Runs out of the box on many platforms like Github pages, Amazon S3, or any web server like Nginx
8. Good developer workflow using Git
I would recommend that you read [good post by David Walsh on advantages and disadvantages of static site generators](https://davidwalsh.name/introduction-static-site-generators). If you look at Google trends, you will notice a steep rise in interest for static site generators. As you can see below, after 2011 more and more people are searching about `static site generator`.
There are many Open-source static site generators options available to the users. You can choose from more than [400 static site generators](https://staticsitegenerators.net/). The most popular ones are [Jekyll](https://github.com/jekyll/jekyll), [Hugo](https://github.com/spf13/hugo), [Middleman](https://github.com/middleman/middleman), [Harp](https://github.com/sintaxi/harp).
> **I have personally used Jekyll and Hugo. I migrated my company blog to Jekyll and it didn't turned out to be a good decision. The two main drawback of Jekyll are 1) you need Ruby runtime 2) it is very slow for bigger projects. To get someone running Jekyll on a machine is a pain. This leads to a barrier in adoption.**
## Why Hugo?
As mentioned above, I had a bad experience with Jekyll so I was looking for an alternative that didn't have same limitations. The reasons I prefer Hugo are:
1. It is not dependent on any programming language runtime
2. It provides binaries for all modern operating system
3. It is Fast
4. It provides quick feedback by live reloading of the content
5. It comes with good defaults and follows convention over configuration philosophy
## Why an online bookshelf?
Some of you might be wondering why I wanted to create an online bookshelf. One promise that I made this year to myself is to read at least one non-technology each month in 2016. Last many years, I am upset with myself for not giving time to read non-technology books. This year I have to change this so I decided to build a bookshelf that will keep track of all the books I read. I got inspired to build my online bookshelf after visiting [Bill Gates blog](https://www.gatesnotes.com/). Bill Gates is maintaining an [awesome bookshelf](https://www.gatesnotes.com/Books) where he shares which his recently read books and their reviews. A screenshot of his bookshelf is shown below.
-----
Building our bookshelf
---
Now, that we know about static site generators and Hugo let's start building our bookshelf step by step. By the end of this tutorial, we will have our bookshelf hosted on Github pages and mapped to a domain.
## Github repository
The code for today’s demo application is available on github: [bookshelf](./bookshelf).
## Step 1: Getting started with Hugo
Go to [https://github.com/spf13/hugo/releases](https://github.com/spf13/hugo/releases) and download Hugo for your operating system. If you are on Mac, the you can install using `brew` package manager as well.
```bash
$ brew update && brew install hugo
```
Once `hugo` is installed, make sure to run the `help` command to verify `hugo` installation. Below I am only showing part of the output of the `help` command for brevity.
```bash
$ hugo help
```
```
hugo is the main command, used to build your Hugo site.
Hugo is a Fast and Flexible Static Site Generator
built with love by spf13 and friends in Go.
Complete documentation is available at http://gohugo.io/.
```
You can check `hugo` version using the command shown below.
```bash
$ hugo version
```
```
Hugo Static Site Generator v0.15 BuildDate: 2015-11-26T11:59:00+05:30
```
> **In this post, we will use the latest version of hugo i.e. version 0.15**
## Step 2: Scaffold bookshelf hugo site
Hugo has commands that allows us to quickly scaffold a Hugo managed website. Navigate to a convenient location on your filesystem and create a new Hugo site `bookshelf` by executing the following command.
```bash
$ hugo new site bookshelf
```
Change directory to `bookshelf` and you will see the following directory layout.
```bash
$ tree -a
```
```
.
|-- archetypes
|-- config.toml
|-- content
|-- data
|-- layouts
`-- static
5 directories, 1 file
```
As mentioned in the command output, `bookshelf` directory has 5 sub-directories and 1 file. Let's look at each of them one by one.
* **archetypes**: You can create new content files in Hugo using the `hugo new` command. When you run that command, it adds few configuration properties to the post like date and title. [Archetype](https://gohugo.io/content/archetypes/) allows you to define your own configuration properties that will be added to the post front matter whenever `hugo new` command is used.
* **config.toml**: Every website should have a configuration file at the root. By default, the configuration file uses `TOML` format but you can also use `YAML` or `JSON` formats as well. [TOML](https://github.com/toml-lang/toml) is minimal configuration file format that's easy to read due to obvious semantics. The configuration settings mentioned in the `config.toml` are applied to the full site. These configuration settings include `baseurl` and `title` of the website.
* **content**: This is where you will store content of the website. Inside content, you will create sub-directories for different sections. Let's suppose your website has three actions -- `blog`, `article`, and `tutorial` then you will have three different directories for each of them inside the `content` directory. The name of the section i.e. `blog`, `article`, or `tutorial` will be used by Hugo to apply a specific layout applicable to that section.
* **data**: This directory is used to store configuration YAML, JSON,or TOML files that can be used by Hugo when generating your website.
* **layouts**: The content inside this directory is used to specify how your content will be converted into the static website.
* **static**: This directory is used to store all the static content that your website will need like images, CSS, JavaScript or other static content.
## Step 3: Add content
Let's now add a post to our `bookshelf`. We will use the `hugo new` command to add a post. In January, I read [Good To Great](http://www.amazon.com/Good-Great-Some-Companies-Others/dp/0066620996/) book so we will start with creating a post for it. **Make sure you are inside the `bookshelf` directory.**
```bash
$ hugo new post/good-to-great.md
```
```
/Users/shekhargulati/bookshelf/content/post/good-to-great.md created
```
The above command will create a new directory `post` inside the `content` directory and create `good-to-great.md` file inside it.
```bash
$ tree -a content
```
```
content
`-- post
`-- good-to-great.md
1 directory, 1 file
```
The content inside the `good-to-great.md` looks like as shown below.
```
+++
date = "2016-02-14T16:11:58+05:30"
draft = true
title = "good to great"
+++
```
The content inside `+++` is the TOML configuration for the post. This configuration is called **front matter**. It enables you to define about the post along with the content. Every post has three configuration properties shown above.
* **date** specifies the date and time at which post was created.
* **draft** specifies that post is not ready for publication yet so it will not be in the generated site
* **title** specifies title for the post
Let's add a small review for **Good to Great** book.
```
+++
date = "2016-02-14T16:11:58+05:30"
draft = true
title = "Good to Great Book Review"
+++
I read **Good to Great in January 2016**. An awesome read sharing detailed analysis on how good companies became great. Although this book is about how companies became great but we could apply a lot of the learnings on ourselves. Concepts like level 5 leader, hedgehog concept, the stockdale paradox are equally applicable to individuals.
```
## Step 4: Serve content
Hugo has inbuilt server that can serve content so that you can preview it. You can also use the inbuilt Hugo server in production as well. To serve content, execute the following command.
```bash
$ hugo server
```
```
0 of 1 draft rendered
0 future content
0 pages created
0 paginator pages created
0 tags created
0 categories created
in 9 ms
Watching for changes in /Users/shekhargulati/bookshelf/{data,content,layouts,static}
Serving pages from memory
Web Server is available at http://localhost:1313/ (bind address 127.0.0.1)
Press Ctrl+C to stop
```
This will start the server on port `1313`. You can view your blog at http://localhost:1313/. When you will go to the link, you will see nothing. There are couple of reasons for that:
1. As you can see in the `hugo server` command output, Hugo didn't rendered the draft. Hugo will only render drafts if you pass `buildDrafts` flag to the `hugo server` command.
2. We have not specified how Markdown content should be rendered. We have to specify a theme that Hugo can use. We will do that in next step.
To render drafts, re-run the server with command shown below.
```bash
$ hugo server --buildDrafts
```
```
1 of 1 draft rendered
0 future content
1 pages created
0 paginator pages created
0 tags created
0 categories created
in 6 ms
Watching for changes in /Users/shekhargulati/bookshelf/{data,content,layouts,static}
Serving pages from memory
Web Server is available at http://localhost:1313/ (bind address 127.0.0.1)
Press Ctrl+C to stop
```
If you go to [http://localhost:1313/](http://localhost:1313/), you will still not view anything as we have not specified theme that Hugo should use.
## Step 5: Add theme
Themes provide the layout and templates that will be used by Hugo to render your website. There are a lot of Open-source themes available at [https://themes.gohugo.io/](https://themes.gohugo.io/) that you can use. From the [Hugo docs](https://gohugo.io/themes/overview/),
> **Hugo currently doesn’t ship with a `default` theme, allowing the user to pick whichever theme best suits their project.**
Themes should be added in the `themes` directory inside the website root. Create new directory themes and change directory to it.
```bash
$ mkdir themes && cd themes
```
Now, you clone one or more themes inside the `themes` directory. We will use robust theme.
```bash
$ git clone git@github.com:dim0627/hugo_theme_robust.git
```
Start the server again
```bash
$ hugo server --theme=hugo_theme_robust --buildDrafts
```
```
1 of 1 draft rendered
0 future content
1 pages created
2 paginator pages created
0 tags created
0 categories created
in 10 ms
Watching for changes in /Users/shekhargulati/bookshelf/{data,content,layouts,static,themes}
Serving pages from memory
Web Server is available at http://localhost:1313/ (bind address 127.0.0.1)
Press Ctrl+C to stop
```
> ** If Hugo will not find a specific theme in the `themes` directory then it will throw an exception as shown below.**
```
FATAL: 2016/02/14 Unable to find theme Directory: /Users/shekhargulati/bookshelf/themes/robust
```
To view your website, you can go to http://localhost:1313/. You will see as shown below.
Let's understand the layout of a theme. A theme consists of following:
* **theme.toml** is the theme configuration file that gives information about the theme like name and description of theme, author details, theme license.
* **images** directory contains two images -- `screenshot.png` and `tn.png`. `screenshot.png` is the image of the list view and `tn.png` is the single post view.
* **layouts** directory contains different views for different content types. Every content type should have two files single.html and list.html. single.html is used for rendering single piece of content. list.html is used to view a list of content items for example all posts with `programming` tag.
* **static** directory stores all the static assets used by the template. This could JavaScript libraries like jQuery or CSS styles or images or any other static content. This directory will be copied into the final site when rendered.
## Step 6: Use multiple themes
You can very easy test different layouts by switching between different themes. Let's suppose we want to try out `bleak` theme. We clone `bleak` theme inside the `themes` directory.
```bash
$ git clone git@github.com:Zenithar/hugo-theme-bleak.git
```
Restart the server using `hugo-theme-bleak`.
```bash
$ hugo server --theme=hugo-theme-bleak --buildDrafts
```
Now, website will use `bleak` theme and will be rendered differently as shown below.
## Step 7: Update config.toml and live reloading in action
Restart the server with `robust` theme as we will use it in this blog.
```bash
$ hugo server --theme=hugo_theme_robust --buildDrafts
```
The website uses the dummy values specified in the `config.toml`. Let's update the configuration.
```toml
baseurl = "http://replace-this-with-your-hugo-site.com/"
languageCode = "en-us"
title = "Shekhar Gulati Book Reviews"
[Params]
Author = "Shekhar Gulati"
```
Hugo has inbuilt support for live reloading. So, as soon as you save your changes it will apply the change and reload the web page. You will see changes as shown below.
The same is reflected in the Hugo server logs as well. As soon as the configuration is changed, it applied the changes.
```
Config file changed: /Users/shekhargulati/bookshelf/config.toml
1 of 1 draft rendered
0 future content
1 pages created
2 paginator pages created
0 tags created
0 categories created
in 11 ms
```
## Step 8: Customize robust theme
Robust theme is a good start towards our online bookshelf but we to customize it a bit to meet the look and feel required for the bookshelf. Hugo makes it very easy to customize themes. You can also create your themes but we will not do that today. If you want to create your own theme, then you should refer to the [Hugo documentation](https://gohugo.io/themes/creation/).
The first change that we have to make is to use a different default image instead of the one used in the theme. The default image used in both the list and single view page resides inside the `themes/hugo_theme_robust/static/images/default.jpg`. We can easily replace it by creating a simple directory structure inside the `static` directory inside the `bookshelf` directory.
Create images directory inside the static directory and copy an image with name `default.jpg` inside it. We will use the default image shown below.
Hugo will sync the changes and reload the website to use new image as shown below.
Now, we need to change the layout of the index page so that only images are shown instead of the text. The index.html inside the layouts directory of the theme refer to partial `li` that renders the list view shown below.
```html
Next, we want to remove information related to theme from the footer. So, create a new file inside the `partials/default_foot.html` with the content copied from the theme `partials/default_foot.html`. Replace the footer section with the one shown below.
```html
```
We also have to remove the sidebar on the right. Copy the index.html from the themes layout directory to the bookshelf layouts directory. Remove the section related to sidebar from the html.
```html
{{ partial "sidebar.html" . }}
```
So far we are using the default image but we would like to use the book image so that we can relate to the book. Every book review will define a configuration setting in its front matter. Update the `good-to-great.md` as shown below.
```
+++
date = "2016-02-14T16:11:58+05:30"
draft = true
title = "Good to Great Book Review"
image = "good-to-great.jpg"
+++
I read **Good to Great in January 2016**. An awesome read sharing detailed analysis on how good companies became great. Although this book is about how companies became great but we could apply a lot of the learnings on ourselves. Concepts like level 5 leader, hedgehog concept, the stockdale paradox are equally applicable to individuals.
```
After adding few more books to our shelf, the shelf looks like as shown below. These are few books that I have read within last one year.
## Step 9: Make posts public
So far all the posts that we have written are in draft status. To make a draft public, you can either run a command or manually change the draft status in the post to True.
```bash
$ hugo undraft content/post/good-to-great.md
```
Now, you can start the server without `buildDrafts` option.
```
$ hugo server --theme=hugo_theme_robust
```
## Step 10: Integrate Disqus
Disqus allows you to integrate comments in your static blog. To enable Disqus, you just have to set `disqusShortname` in the config.toml as shown below.
```
[Params]
Author = "Shekhar Gulati"
disqusShortname = "shekhargulati"
```
Now, commenting will be enabled in your blog.
## Step 11: Generate website
To generate Hugo website code that you can use to deploy your website, type the following command.
```bash
$ hugo --theme=hugo_theme_robust
0 draft content
0 future content
5 pages created
2 paginator pages created
0 tags created
0 categories created
in 17 ms
```
> **Make sure to change the baseurl. For my bookshelf on Github pages, url is [https://shekhargulati.github.io/bookshelf](https://shekhargulati.github.io/bookshelf)**
After you run the hugo command, a public directory will be created with the generated website source.
## Step 12: Deploy bookshelf on Github pages
Create a new repository with name `bookshelf` on Github. Once created, create a new Git repo on local system and add remote.
```bash
$ mkdir bookshelf-public
$ cd bookshelf-public
$ git init
$ git remote add origin git@github.com:shekhargulati/bookshelf.git
```
Copy the content of the `public` directory to the `bookshelf-public` directory. Run this command from with in the `bookshelf-public` directory.
```bash
$ cp -r ../bookshelf/public/ .
```
Create new branch `gh-pages` and checkout it.
```bash
$ git checkout -b gh-pages
Switched to a new branch 'gh-pages'
```
Add all the files to the index, commit them, and push the changes to Github.
```bash
$ git add --all
$ git commit -am "bookshelf added"
$ git push origin gh-pages
```
In couple of minutes, your website will be live https://shekhargulati.github.io/bookshelf/.
----
That's all for this week. Please provide your valuable feedback by adding a comment to [https://github.com/shekhargulati/52-technologies-in-2016/issues/10](https://github.com/shekhargulati/52-technologies-in-2016/issues/10).
[](https://github.com/igrigorik/ga-beacon)
================================================
FILE: 07-hugo/bookshelf/config.toml
================================================
baseurl = "http://example.com"
languageCode = "en-us"
title = "Shekhar Gulati Bookshelf"
[Params]
Author = "Shekhar Gulati"
disqusShortname = "shekhargulati"
GoogleAnalyticsUserID = "UA-73784822-1"
================================================
FILE: 07-hugo/bookshelf/content/post/art-of-thinking-clearly.md
================================================
+++
date = "2016-02-14T19:10:29+05:30"
draft = false
title = "art of thinking clearly"
image = "art-of-thinking-clearly.jpg"
+++
================================================
FILE: 07-hugo/bookshelf/content/post/confessions-of-a-public-speaker.md
================================================
+++
date = "2016-02-14T19:10:48+05:30"
draft = false
title = "confessions of a public speaker"
image = "confessions-of-a-public-speaker.png"
+++
================================================
FILE: 07-hugo/bookshelf/content/post/good-to-great.md
================================================
+++
date = "2016-02-14T19:23:40+05:30"
draft = false
title = "Good to Great Book Review"
image = "good-to-great.jpg"
+++
I read **Good to Great in January 2016**. An awesome read sharing detailed analysis on how good companies became great. Although this book is about how companies became great but we could apply a lot of the learnings on ourselves. Concepts like level 5 leader, hedgehog concept, the stockdale paradox are equally applicable to individuals.
================================================
FILE: 07-hugo/bookshelf/content/post/hen-who-dreamed-she-could-fly.md
================================================
+++
date = "2016-02-14T19:06:04+05:30"
draft = false
title = "Hen Who Dreamed She Could Fly"
image = "hen-who-dreamed.jpg"
+++
In February 2016, I read **The Hen Who Dreamed She Could Fly**. This short book is an amazing and inspiring story of a hen Sprout who wanted to live a normal happy life outside of a coop. The lesson I learnt from this book is that you have to break shackles and work towards a life that you want to live. If you believe in your dreams and don’t give up then you will live a life worth living.
================================================
FILE: 07-hugo/bookshelf/content/post/seven-habbits-of-highly-effective-people.md
================================================
+++
date = "2016-02-14T19:11:05+05:30"
draft = false
title = "seven habbits of highly effective people"
image = "seven-habits-of-highly-effective-people.jpg"
+++
================================================
FILE: 07-hugo/bookshelf/layouts/_default/li.html
================================================
{{ partial "default_head.html" . }}
{{ partial "pagination.html" . }}
{{ partial "default_foot.html" . }}
================================================
FILE: 07-hugo/bookshelf/layouts/partials/default_foot.html
================================================
{{ with .Site.Params.GoogleAnalyticsUserID }}
{{ end }}
{{ range $key, $value := .Paginator.Pages }}
{{ .Render "li" }}
{{ end }}