Repository: Spinoco/fs2-http
Branch: series/0.4
Commit: 8f8ef9a139bb
Files: 45
Total size: 163.5 KB
Directory structure:
gitextract_8b82oz4x/
├── .gitignore
├── .travis.yml
├── LICENSE
├── README.md
├── build.sbt
├── doc/
│ └── custom_codec.md
├── project/
│ ├── build.properties
│ └── plugins.sbt
├── sbt
├── src/
│ ├── main/
│ │ └── scala/
│ │ └── spinoco/
│ │ └── fs2/
│ │ └── http/
│ │ ├── HttpClient.scala
│ │ ├── HttpRequestOrResponse.scala
│ │ ├── HttpServer.scala
│ │ ├── body/
│ │ │ ├── BodyDecoder.scala
│ │ │ ├── BodyEncoder.scala
│ │ │ ├── StreamBodyDecoder.scala
│ │ │ └── StreamBodyEncoder.scala
│ │ ├── http.scala
│ │ ├── internal/
│ │ │ ├── ChunkedEncoding.scala
│ │ │ └── internal.scala
│ │ ├── routing/
│ │ │ ├── MatchResult.scala
│ │ │ ├── Matcher.scala
│ │ │ ├── StringDecoder.scala
│ │ │ └── routing.scala
│ │ ├── sse/
│ │ │ ├── SSEDecoder.scala
│ │ │ ├── SSEEncoder.scala
│ │ │ ├── SSEEncoding.scala
│ │ │ └── SSEMessage.scala
│ │ ├── util/
│ │ │ └── util.scala
│ │ └── websocket/
│ │ ├── Frame.scala
│ │ ├── WebSocket.scala
│ │ ├── WebSocketRequest.scala
│ │ └── package.scala
│ └── test/
│ └── scala/
│ └── spinoco/
│ └── fs2/
│ └── http/
│ ├── HttpRequestSpec.scala
│ ├── HttpResponseSpec.scala
│ ├── HttpServerSpec.scala
│ ├── Resources.scala
│ ├── internal/
│ │ ├── ChunkedEncodingSpec.scala
│ │ ├── HttpClientApp.scala
│ │ └── HttpServerApp.scala
│ ├── routing/
│ │ └── MatcherSpec.scala
│ ├── sse/
│ │ └── SSEEncodingSpec.scala
│ ├── util/
│ │ └── UtilSpec.scala
│ └── websocket/
│ ├── WebSocketClientApp.scala
│ └── WebSocketSpec.scala
└── version.sbt
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
*.class
*.log
# sbt specific
.cache
.history
.lib/
dist/*
target/
lib_managed/
src_managed/
project/boot/
project/plugins/project/
# Scala-IDE specific
.scala_dependencies
.worksheet
.idea/*
.DS_Store
================================================
FILE: .travis.yml
================================================
language : scala
scala:
- 2.11.12
- 2.12.6
cache:
directories:
- $HOME/.ivy2
- $HOME/.sbt
jdk:
- oraclejdk8
script:
- sbt ++$TRAVIS_SCALA_VERSION -Dfile.encoding=UTF8 "project fs2-http" test
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2017 Spinoco
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: README.md
================================================
# fs2-http
Minimalistic yet powerful http client and server with scala fs2 library.
[](https://travis-ci.org/Spinoco/fs2-http)
[](https://gitter.im/fs2-http/Lobby)
## Overview
fs2-http is a simple client and server library that allows you to build http clients and servers using scala fs2.
The aim of fs2-http is to provide simple and reusable components that enable fast work with various
http protocols.
All the code is fully asynchronous and non-blocking. Thanks to fs2, this comes with back-pressure and streaming support.
fs2-http was built by compiling the internal projects Spinoco uses for building its [product](http://www.spinoco.com/), where the server side is completely implemented in fs2.
Currently the project only has three dependencies: fs2, scodec and shapeless. As such you are free to use this with any other
functional library, such as scalaz or cats.
### Features
- HTTP 1.1 Client (request/reply, websocket, server-side-events) with SSL support
- HTTP 1.1 Server (request/reply, routing, websocket, server-side-events)
- HTTP Chunked encoding
### SBT
Add this to your sbt build file :
```
libraryDependencies += "com.spinoco" %% "fs2-http" % "0.4.0"
```
### Dependencies
version | scala | fs2 | scodec | shapeless
---------|-----------|--------|---------|-----------
0.4.0 | 2.11, 2.12| 1.0.0 | 1.10.3 | 2.3.2
## Usage
Throughout this usage guide, the following imports are required in order for you to be able to run the examples test:console:
```
import fs2._
import fs2.util.syntax._
import cats.effect._
import cats.syntax.all._
import spinoco.fs2.http
import http._
import http.websocket._
import spinoco.protocol.http.header._
import spinoco.protocol.http._
import spinoco.protocol.http.header.value._
// import resources (Executor, Strategy, Asynchronous Channel Group, ...)
import spinoco.fs2.http.Resources._
```
### HTTP Client
Currently fs2-http supports HTTP 1.1 protocol and allows you to connect to server with either http:// or https:// scheme.
A simple client that requests https page body data with the GET method from `https://github.com/Spinoco/fs2-http` may be constructed, for example, as:
```
http.client[IO]().flatMap { client =>
val request = HttpRequest.get[IO](Uri.https("github.com", "/Spinoco/fs2-http"))
client.request(request).flatMap { resp =>
Stream.eval(resp.bodyAsString)
}.runLog.map {
println
}
}.unsafeRunSync()
```
The above code snippet only "builds" the http client, resulting in `IO` that will be evaluated once run (using `unsafeRunSync()`).
The line with `Stream.eval(resp.bodyAsString)` on it actually evaluates the consumed body of the response. The body of the
response can be evaluated strictly (meaning all output is first collected and then converted to the desired type), or it can be streamed (meaning it will be converted to the desired type as it is received from the server). A streamed body is accessible as `resp.body`.
Requests to the server are modeled with [HttpRequest\[F\]](https://github.com/Spinoco/fs2-http/blob/master/src/main/scala/spinoco/fs2/http/HttpRequestOrResponse.scala#L116), and responses are modeled as [HttpResponse\[F\]](https://github.com/Spinoco/fs2-http/blob/master/src/main/scala/spinoco/fs2/http/HttpRequestOrResponse.scala#L232). Both of them share several [helpers](https://github.com/Spinoco/fs2-http/blob/master/src/main/scala/spinoco/fs2/http/HttpRequestOrResponse.scala#L17) to help you work easily with the body.
There is also a simple way to sent (stream) arbitrary data to server. It is easily achieved by modifying the request accordingly:
```
val stringStream: Stream[IO, String] = ???
implicit val encoder = StreamBodyEncoder.utf8StringEncoder[IO]
HttpRequest.get(Uri.https("github.com", "/Spinoco/fs2-http"))
.withMethod(HttpMethod.POST)
.withStreamBody(stringStream)
```
In the example above the request is build as such, to ensure that when run by the client it will consume `stringStream` and send it with PUT request as utf8 encoded body to server.
### WebSocket
fs2-http has support for websocket clients (RFC 6455). A websocket client is built with the following construct:
```
def wsPipe: Pipe[IO, Frame[String], Frame[String]] = { inbound =>
val output = time.awakeEvery[IO](1.second).map { dur => println(s"SENT $dur"); Frame.Text(s" ECHO $dur") }.take(5)
output.concurrently(inbound.take(5).map { in => println(("RECEIVED ", in)) })
}
http.client[IO]().flatMap { client =>
val request = WebSocketRequest.ws("echo.websocket.org", "/", "encoding" -> "text")
client.websocket(request, wsPipe).run
}.unsafeRun()
```
The above code will create a pipe that receives websocket frames and expects the server to echo them back. As you see,
there is no direct access to response or body, instead websockets are always supplied with fs2 `Pipe` to send and receive data.
This is in fact quite a powerful construct that allows you to asynchronously send and receive data to/from server over http/https with full back-pressure support.
Websockets use `Frame[A]` to send and receive data. Frame is used to tag a given frame as binary or text. To encode/decode `A` the `scodec.Encoder` and `scodec.Decoder` is used.
### HTTP Server
fs2-http supports building simple yet fully functional HTTP servers. The following construct builds a very simple echo server:
```
import java.net.InetSocketAddress
import java.util.concurrent.Executors
import java.nio.channels.AsynchronousChannelGroup
val ES = Executors.newCachedThreadPool(Strategy.daemonThreadFactory("ACG"))
implicit val ACG = AsynchronousChannelGroup.withThreadPool(ES) // http.server requires a group
implicit val S = Strategy.fromExecutor(ES) // Async (Task) requires a strategy
def service(request: HttpRequestHeader, body: Stream[IO,Byte]): Stream[IO,HttpResponse[IO]] = {
if (request.path != Uri.Path / "echo") Stream.emit(HttpResponse(HttpStatusCode.Ok).withUtf8Body("Hello World"))
else {
val ct = request.headers.collectFirst { case `Content-Type`(ct) => ct }.getOrElse(ContentType(MediaType.`application/octet-stream`, None, None))
val size = request.headers.collectFirst { case `Content-Length`(sz) => sz }.getOrElse(0l)
val ok = HttpResponse(HttpStatusCode.Ok).chunkedEncoding.withContentType(ct).withBodySize(size)
Stream.emit(ok.copy(body = body.take(size)))
}
}
http.server(new InetSocketAddress("127.0.0.1", 9090))(service).run.unsafeRun()
```
As you see the server creates a simple `Stream[F,Unit]` that, when run, will bind itself to 127.0.0.1 port 9090 and will serve the results of the `service` function.
The service function is defined as `(HttpRequestHeader, Stream[F, Body]) => Stream[F, HttpResponse[F]` and allows you to perform arbitrary functionality, all wrapped in `fs2.Stream`.
Writing a server service function manually may not be fun and may result in unreadable and hard to maintain code. As such the last component of fs2-http is server routing.
### HTTP Server Routing
Server routing is a micro-dsl language to allow fast monadic composition of a parser, that is essentially a function `(HttpRequestHeader, Stream[F, Body]) => Either[HttpResponse[F], Stream[F, HttpResponse[F]]`
where on the right side there is the result when the parser matches, and on the left side there is the response when the parser fails to match.
Thanks to the parser's ability to compose, you can build quite complex routing constructs, that remain readable:
```
import spinoco.fs2.http.routing._
import shapeless.{HNil, ::}
route[IO] ( choice(
"example1" / "path" map { case _ => ??? }
, "example2" / as[Int] :/: as[String] map { case int :: s :: HNil => ??? }
, "example3" / body.as[Foo] :: choice(Post, Put) map { case foo :: postOrPut :: HNil => ??? }
, "example4" / header[`Content-Type`] map { case contentType => ??? }
, "example5" / param[Int]("count") :: param[String]("query") map { case count :: query :: HNil => ??? }
, "example6" / eval(someEffect) map { case result => ??? }
))
```
Here the choice indicates that any of the supplied routes may match, starting with the very first route. Instead of ??? you may supply any function producing the `Stream[IO, HttpResponse[IO]]`, that will be evaluated when the route will match.
The meaning of the individual routes is as follows:
- example1 : will match path "/example1/path"
- example2 : will match path "/example2/23/some_string" and will produce 23 :: "some_string" :: HNil to map
- example3 : will match path "/example3" and will consume body to produce `Foo` class. Map is supplied with Foo :: HttpMethod.Value :: HNil
- example4 : will match path "/example4" and will match if header `Content-Type` is present supplying that header to map.
- example5 : will match path "/example5?count=1&query=sql_query" supplying 1 :: "sql:query" :: HNil to map
- example6 : will match path "/example6" and then evaluating `someEffect` where the result of someEffect will be passed to map
### Other documentation and helpful links
- [Using custom headers](https://github.com/Spinoco/fs2-http/blob/master/doc/custom_codec.md)
### Comparing to http://http4s.org/
Http4s.org is a very useful library for http, originally started with scalaz-stream and currently fully supporting fs2.
The main differences between http4s.org and fs2-http is that unlike http4s.org, fs2-http is purely functional, including the network stack
which is completely impliemented in fs2. Also the fs2-http focuses to be minimalistic both on dependencies and functionality provided.
================================================
FILE: build.sbt
================================================
import com.typesafe.sbt.pgp.PgpKeys.publishSigned
val ReleaseTag = """^release/([\d\.]+a?)$""".r
lazy val contributors = Seq(
"pchlupacek" -> "Pavel Chlupáček"
)
lazy val commonSettings = Seq(
organization := "com.spinoco",
scalaVersion := "2.12.6",
crossScalaVersions := Seq("2.11.12", "2.12.6"),
scalacOptions ++= Seq(
"-feature",
"-deprecation",
"-language:implicitConversions",
"-language:higherKinds",
"-language:existentials",
"-language:postfixOps",
"-Xfatal-warnings",
"-Yno-adapted-args",
"-Ywarn-value-discard",
"-Ywarn-unused-import"
),
scalacOptions in (Compile, console) ~= {_.filterNot("-Ywarn-unused-import" == _)},
scalacOptions in (Test, console) := (scalacOptions in (Compile, console)).value,
libraryDependencies ++= Seq(
compilerPlugin("org.scalamacros" % "paradise" % "2.1.0" cross CrossVersion.full)
, "org.scodec" %% "scodec-bits" % "1.1.4"
, "org.scodec" %% "scodec-core" % "1.10.3"
, "com.spinoco" %% "protocol-http" % "0.3.15"
, "com.spinoco" %% "protocol-websocket" % "0.3.15"
, "co.fs2" %% "fs2-core" % "1.0.0"
, "co.fs2" %% "fs2-io" % "1.0.0"
, "com.spinoco" %% "fs2-crypto" % "0.4.0"
, "org.scalacheck" %% "scalacheck" % "1.13.4" % "test"
),
scmInfo := Some(ScmInfo(url("https://github.com/Spinoco/fs2-http"), "git@github.com:Spinoco/fs2-http.git")),
homepage := None,
licenses += ("MIT", url("http://opensource.org/licenses/MIT")),
initialCommands := s"""
import fs2._
import fs2.util.syntax._
import spinoco.fs2.http
import http.Resources._
import spinoco.protocol.http.header._
"""
) ++ testSettings ++ scaladocSettings ++ publishingSettings ++ releaseSettings
lazy val testSettings = Seq(
parallelExecution in Test := false,
testOptions in Test += Tests.Argument(TestFrameworks.ScalaTest, "-oDF"),
publishArtifact in Test := true
)
lazy val scaladocSettings = Seq(
scalacOptions in (Compile, doc) ++= Seq(
"-doc-source-url", scmInfo.value.get.browseUrl + "/tree/master€{FILE_PATH}.scala",
"-sourcepath", baseDirectory.in(LocalRootProject).value.getAbsolutePath,
"-implicits",
"-implicits-show-all"
),
scalacOptions in (Compile, doc) ~= { _ filterNot { _ == "-Xfatal-warnings" } },
autoAPIMappings := true
)
lazy val publishingSettings = Seq(
publishTo := {
val nexus = "https://oss.sonatype.org/"
if (version.value.trim.endsWith("SNAPSHOT"))
Some("snapshots" at nexus + "content/repositories/snapshots")
else
Some("releases" at nexus + "service/local/staging/deploy/maven2")
},
credentials ++= (for {
username <- Option(System.getenv().get("SONATYPE_USERNAME"))
password <- Option(System.getenv().get("SONATYPE_PASSWORD"))
} yield Credentials("Sonatype Nexus Repository Manager", "oss.sonatype.org", username, password)).toSeq,
publishMavenStyle := true,
pomIncludeRepository := { _ => false },
pomExtra := {
<url>https://github.com/Spinoco/fs2-http</url>
<developers>
{for ((username, name) <- contributors) yield
<developer>
<id>{username}</id>
<name>{name}</name>
<url>http://github.com/{username}</url>
</developer>
}
</developers>
},
pomPostProcess := { node =>
import scala.xml._
import scala.xml.transform._
def stripIf(f: Node => Boolean) = new RewriteRule {
override def transform(n: Node) =
if (f(n)) NodeSeq.Empty else n
}
val stripTestScope = stripIf { n => n.label == "dependency" && (n \ "scope").text == "test" }
new RuleTransformer(stripTestScope).transform(node)(0)
}
, resolvers += Resolver.mavenLocal
)
lazy val releaseSettings = Seq(
releaseCrossBuild := true,
releasePublishArtifactsAction := PgpKeys.publishSigned.value
)
lazy val `fs2-http`=
project.in(file("./"))
.settings(commonSettings)
.settings(
name := "fs2-http"
)
================================================
FILE: doc/custom_codec.md
================================================
# Using custom codec for http headers and requests.
Ocassionally it is required to extends headers supported by fs2-http by some custom headers of user choice. Behind the scenes fs2-http is using scodec library for encoding and decoding codecs. So generally addin any codec is quite straigthforward.
## Using custom Generic Header
If you are ok with receiveving your header as simple String value pair, there is simple technique using the `GenericHeader`. This allows you to encode and decode any Http Header with simple string key and value pair, where key is name of the header and value is anything after : in http header. For example :
```
Authorization: Token someAuthorizationToken
```
may be decoded as
```scala
GenericHeader("Authorization", "Token someAuthorizationToken")
```
However to do so we need to supply this codec to the http client and http server. In both cases this is pretty straightforward to do:
```scala
import spinoco.protocol.http
import spinoco.protocol.http.codec.HttpHeaderCodec
val genericHeaderAuthCodec: HttpCodec[HttpHeader] =
utf8.xmap[GenericHeader](s => GenericHeader("Authorization", s), _.value).upcast[HttpHeader]
val headerCodec: Codec[HttpHeader]=
HttpHeaderCodec.codec(Int.MaxValue, ("Authorization" -> genericHeaderAuthCodec))
http.client(
requestCodec = HttpRequestHeaderCodec.codec(headerCodec)
, responseCodec = HttpResponseHeaderCodec.codec(headerCodec)
) map { client =>
/** your code with client **/
}
http.server(
bindTo = ??? // your ip where you want bind server to
, requestCodec = HttpRequestHeaderCodec.codec(headerCodec)
, responseCodec = HttpResponseHeaderCodec.codec(headerCodec)
) flatMap { server =>
/** your code with server **/
}
```
Note that this technique, effectivelly causes to turn-off any already supported Authorization header codecs, which you man not want to. Well, in next section we describe exactly solution for that.
## Using custom header codec
Custom header codecs allow you to write any header codec available or extends it by your own functionality. So lets say we would like to extend Authorization header with our own version of Authorization header while still keeping the current Authroization header codec in place.
Let's say we ahve our own Authorization header case class :
```scala
case class MyAuthorizationTokenHeader(token: String) extends HttpHeader
```
First we need to create codec that will encode authorization of our own, and then, when that won't pass, we will try to decode with default. This is quite simply achievable by following code snipped: {
```scala
import scodec.codecs._
import spinoco.protocol.http.codec.helper._
import spinoco.procotol.http.header.value.Authorization
object MyAuthorizationTokenHeader {
// this is simple codec to decode essentially line `Token sometoken`
val codec : Codec[MyAuthorizationTokenHeader) =
(asciiConstant("Token") ~> (whitespace() ~> utf8String)).xmap(
{ token => MyAuthorizationTokenHeader(token)}
, _.token
)
// this is new codec, that will first try to decode by our codec and then if that fails, will use default authroization codec.
val customAuthorizationHeader: Codec[HttpHeader] = choice(
codec.upcast[HttpHeader]
, Authroization.codec
)
}
```
Once we have that custom codec setup, we only need to plug it to client and/or server likewise we did for GenericHeader before. For example:
```scala
import spinoco.protocol.http
import spinoco.protocol.http.codec.HttpHeaderCodec
val headerCodec: Codec[HttpHeader]=
HttpHeaderCodec.codec(Int.MaxValue, ("Authorization" -> MyAuthorizationTokenHeader.customAuthorizationHeader))
http.client(
requestCodec = HttpRequestHeaderCodec.codec(headerCodec)
, responseCodec = HttpResponseHeaderCodec.codec(headerCodec)
) map { client =>
/** your code with client **/
}
http.server(
bindTo = ??? // your ip where you want bind server to
, requestCodec = HttpRequestHeaderCodec.codec(headerCodec)
, responseCodec = HttpResponseHeaderCodec.codec(headerCodec)
) flatMap { server =>
/** your code with server **/
}
```
## Summary
Both of these techniques have own advantages and drawbacks. It is up to user to decide whichever suits best. However, as you may see with a little effort you may plug very complex encoding and decoding schemes (even including any binary data) that your application may require.
================================================
FILE: project/build.properties
================================================
sbt.version=1.1.6
================================================
FILE: project/plugins.sbt
================================================
addSbtPlugin("com.github.tkawachi" % "sbt-doctest" % "0.8.0")
addSbtPlugin("com.github.gseitz" % "sbt-release" % "1.0.9")
addSbtPlugin("com.jsuereth" % "sbt-pgp" % "1.1.2")
addSbtPlugin("org.xerial.sbt" % "sbt-sonatype" % "2.3")
================================================
FILE: sbt
================================================
#!/usr/bin/env bash
#
# A more capable sbt runner, coincidentally also called sbt.
# Author: Paul Phillips <paulp@improving.org>
set -o pipefail
# todo - make this dynamic
declare -r sbt_release_version="0.13.11"
declare -r sbt_unreleased_version="0.13.11"
declare -r buildProps="project/build.properties"
declare sbt_jar sbt_dir sbt_create sbt_version
declare scala_version sbt_explicit_version
declare verbose noshare batch trace_level log_level
declare sbt_saved_stty debugUs
echoerr () { echo >&2 "$@"; }
vlog () { [[ -n "$verbose" ]] && echoerr "$@"; }
# spaces are possible, e.g. sbt.version = 0.13.0
build_props_sbt () {
[[ -r "$buildProps" ]] && \
grep '^sbt\.version' "$buildProps" | tr '=\r' ' ' | awk '{ print $2; }'
}
update_build_props_sbt () {
local ver="$1"
local old="$(build_props_sbt)"
[[ -r "$buildProps" ]] && [[ "$ver" != "$old" ]] && {
perl -pi -e "s/^sbt\.version\b.*\$/sbt.version=${ver}/" "$buildProps"
grep -q '^sbt.version[ =]' "$buildProps" || printf "\nsbt.version=%s\n" "$ver" >> "$buildProps"
vlog "!!!"
vlog "!!! Updated file $buildProps setting sbt.version to: $ver"
vlog "!!! Previous value was: $old"
vlog "!!!"
}
}
set_sbt_version () {
sbt_version="${sbt_explicit_version:-$(build_props_sbt)}"
[[ -n "$sbt_version" ]] || sbt_version=$sbt_release_version
export sbt_version
}
# restore stty settings (echo in particular)
onSbtRunnerExit() {
[[ -n "$sbt_saved_stty" ]] || return
vlog ""
vlog "restoring stty: $sbt_saved_stty"
stty "$sbt_saved_stty"
unset sbt_saved_stty
}
# save stty and trap exit, to ensure echo is reenabled if we are interrupted.
trap onSbtRunnerExit EXIT
sbt_saved_stty="$(stty -g 2>/dev/null)"
vlog "Saved stty: $sbt_saved_stty"
# this seems to cover the bases on OSX, and someone will
# have to tell me about the others.
get_script_path () {
local path="$1"
[[ -L "$path" ]] || { echo "$path" ; return; }
local target="$(readlink "$path")"
if [[ "${target:0:1}" == "/" ]]; then
echo "$target"
else
echo "${path%/*}/$target"
fi
}
die() {
echo "Aborting: $@"
exit 1
}
url_base () {
local version="$1"
case "$version" in
0.7.*) echo "http://simple-build-tool.googlecode.com" ;;
0.10.* ) echo "$sbt_launch_release_repo" ;;
0.11.[12]) echo "$sbt_launch_release_repo" ;;
*-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]-[0-9][0-9][0-9][0-9][0-9][0-9]) # ie "*-yyyymmdd-hhMMss"
echo "$sbt_launch_snapshot_repo" ;;
*) echo "$sbt_launch_release_repo" ;;
esac
}
make_url () {
local version="$1"
local base="${sbt_launch_repo:-$(url_base "$version")}"
case "$version" in
0.7.*) echo "$base/files/sbt-launch-0.7.7.jar" ;;
0.10.* ) echo "$base/org.scala-tools.sbt/sbt-launch/$version/sbt-launch.jar" ;;
0.11.[12]) echo "$base/org.scala-tools.sbt/sbt-launch/$version/sbt-launch.jar" ;;
*) echo "$base/org.scala-sbt/sbt-launch/$version/sbt-launch.jar" ;;
esac
}
init_default_option_file () {
local overriding_var="${!1}"
local default_file="$2"
if [[ ! -r "$default_file" && "$overriding_var" =~ ^@(.*)$ ]]; then
local envvar_file="${BASH_REMATCH[1]}"
if [[ -r "$envvar_file" ]]; then
default_file="$envvar_file"
fi
fi
echo "$default_file"
}
declare -r cms_opts="-XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC"
declare -r jit_opts="-XX:ReservedCodeCacheSize=256m -XX:+TieredCompilation"
declare -r default_jvm_opts_common="-Xms512m -Xmx1536m -Xss2m $jit_opts $cms_opts"
declare -r noshare_opts="-Dsbt.global.base=project/.sbtboot -Dsbt.boot.directory=project/.boot -Dsbt.ivy.home=project/.ivy"
declare -r latest_28="2.8.2"
declare -r latest_29="2.9.3"
declare -r latest_210="2.10.6"
declare -r latest_211="2.11.8"
declare -r latest_212="2.12.0-M3"
declare -r sbt_launch_release_repo="http://repo.typesafe.com/typesafe/ivy-releases"
declare -r sbt_launch_snapshot_repo="https://repo.scala-sbt.org/scalasbt/ivy-snapshots"
declare -r script_path="$(get_script_path "$BASH_SOURCE")"
declare -r script_name="${script_path##*/}"
# some non-read-onlies set with defaults
declare java_cmd="java"
declare sbt_opts_file="$(init_default_option_file SBT_OPTS .sbtopts)"
declare jvm_opts_file="$(init_default_option_file JVM_OPTS .jvmopts)"
declare sbt_launch_dir="$HOME/.sbt/launchers"
declare sbt_launch_repo
# pull -J and -D options to give to java.
declare -a residual_args
declare -a java_args
declare -a scalac_args
declare -a sbt_commands
# args to jvm/sbt via files or environment variables
declare -a extra_jvm_opts extra_sbt_opts
addJava () {
vlog "[addJava] arg = '$1'"
java_args+=("$1")
}
addSbt () {
vlog "[addSbt] arg = '$1'"
sbt_commands+=("$1")
}
setThisBuild () {
vlog "[addBuild] args = '$@'"
local key="$1" && shift
addSbt "set $key in ThisBuild := $@"
}
addScalac () {
vlog "[addScalac] arg = '$1'"
scalac_args+=("$1")
}
addResidual () {
vlog "[residual] arg = '$1'"
residual_args+=("$1")
}
addResolver () {
addSbt "set resolvers += $1"
}
addDebugger () {
addJava "-Xdebug"
addJava "-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=$1"
}
setScalaVersion () {
[[ "$1" == *"-SNAPSHOT" ]] && addResolver 'Resolver.sonatypeRepo("snapshots")'
addSbt "++ $1"
}
setJavaHome () {
java_cmd="$1/bin/java"
setThisBuild javaHome "scala.Some(file(\"$1\"))"
export JAVA_HOME="$1"
export JDK_HOME="$1"
export PATH="$JAVA_HOME/bin:$PATH"
}
setJavaHomeQuietly () {
addSbt warn
setJavaHome "$1"
addSbt info
}
# if set, use JDK_HOME/JAVA_HOME over java found in path
if [[ -e "$JDK_HOME/lib/tools.jar" ]]; then
setJavaHomeQuietly "$JDK_HOME"
elif [[ -e "$JAVA_HOME/bin/java" ]]; then
setJavaHomeQuietly "$JAVA_HOME"
fi
# directory to store sbt launchers
[[ -d "$sbt_launch_dir" ]] || mkdir -p "$sbt_launch_dir"
[[ -w "$sbt_launch_dir" ]] || sbt_launch_dir="$(mktemp -d -t sbt_extras_launchers.XXXXXX)"
java_version () {
local version=$("$java_cmd" -version 2>&1 | grep -E -e '(java|openjdk) version' | awk '{ print $3 }' | tr -d \")
vlog "Detected Java version: $version"
echo "${version:2:1}"
}
# MaxPermSize critical on pre-8 jvms but incurs noisy warning on 8+
default_jvm_opts () {
local v="$(java_version)"
if [[ $v -ge 8 ]]; then
echo "$default_jvm_opts_common"
else
echo "-XX:MaxPermSize=384m $default_jvm_opts_common"
fi
}
build_props_scala () {
if [[ -r "$buildProps" ]]; then
versionLine="$(grep '^build.scala.versions' "$buildProps")"
versionString="${versionLine##build.scala.versions=}"
echo "${versionString%% .*}"
fi
}
execRunner () {
# print the arguments one to a line, quoting any containing spaces
vlog "# Executing command line:" && {
for arg; do
if [[ -n "$arg" ]]; then
if printf "%s\n" "$arg" | grep -q ' '; then
printf >&2 "\"%s\"\n" "$arg"
else
printf >&2 "%s\n" "$arg"
fi
fi
done
vlog ""
}
[[ -n "$batch" ]] && exec </dev/null
exec "$@"
}
jar_url () {
make_url "$1"
}
jar_file () {
echo "$sbt_launch_dir/$1/sbt-launch.jar"
}
download_url () {
local url="$1"
local jar="$2"
echoerr "Downloading sbt launcher for $sbt_version:"
echoerr " From $url"
echoerr " To $jar"
mkdir -p "${jar%/*}" && {
if which curl >/dev/null; then
curl --fail --silent --location "$url" --output "$jar"
elif which wget >/dev/null; then
wget --quiet -O "$jar" "$url"
fi
} && [[ -r "$jar" ]]
}
acquire_sbt_jar () {
local sbt_url="$(jar_url "$sbt_version")"
sbt_jar="$(jar_file "$sbt_version")"
[[ -r "$sbt_jar" ]] || download_url "$sbt_url" "$sbt_jar"
}
usage () {
set_sbt_version
cat <<EOM
Usage: $script_name [options]
Note that options which are passed along to sbt begin with -- whereas
options to this runner use a single dash. Any sbt command can be scheduled
to run first by prefixing the command with --, so --warn, --error and so on
are not special.
Output filtering: if there is a file in the home directory called .sbtignore
and this is not an interactive sbt session, the file is treated as a list of
bash regular expressions. Output lines which match any regex are not echoed.
One can see exactly which lines would have been suppressed by starting this
runner with the -x option.
-h | -help print this message
-v verbose operation (this runner is chattier)
-d, -w, -q aliases for --debug, --warn, --error (q means quiet)
-x debug this script
-trace <level> display stack traces with a max of <level> frames (default: -1, traces suppressed)
-debug-inc enable debugging log for the incremental compiler
-no-colors disable ANSI color codes
-sbt-create start sbt even if current directory contains no sbt project
-sbt-dir <path> path to global settings/plugins directory (default: ~/.sbt/<version>)
-sbt-boot <path> path to shared boot directory (default: ~/.sbt/boot in 0.11+)
-ivy <path> path to local Ivy repository (default: ~/.ivy2)
-no-share use all local caches; no sharing
-offline put sbt in offline mode
-jvm-debug <port> Turn on JVM debugging, open at the given port.
-batch Disable interactive mode
-prompt <expr> Set the sbt prompt; in expr, 's' is the State and 'e' is Extracted
# sbt version (default: sbt.version from $buildProps if present, otherwise $sbt_release_version)
-sbt-force-latest force the use of the latest release of sbt: $sbt_release_version
-sbt-version <version> use the specified version of sbt (default: $sbt_release_version)
-sbt-dev use the latest pre-release version of sbt: $sbt_unreleased_version
-sbt-jar <path> use the specified jar as the sbt launcher
-sbt-launch-dir <path> directory to hold sbt launchers (default: $sbt_launch_dir)
-sbt-launch-repo <url> repo url for downloading sbt launcher jar (default: $(url_base "$sbt_version"))
# scala version (default: as chosen by sbt)
-28 use $latest_28
-29 use $latest_29
-210 use $latest_210
-211 use $latest_211
-212 use $latest_212
-scala-home <path> use the scala build at the specified directory
-scala-version <version> use the specified version of scala
-binary-version <version> use the specified scala version when searching for dependencies
# java version (default: java from PATH, currently $(java -version 2>&1 | grep version))
-java-home <path> alternate JAVA_HOME
# passing options to the jvm - note it does NOT use JAVA_OPTS due to pollution
# The default set is used if JVM_OPTS is unset and no -jvm-opts file is found
<default> $(default_jvm_opts)
JVM_OPTS environment variable holding either the jvm args directly, or
the reference to a file containing jvm args if given path is prepended by '@' (e.g. '@/etc/jvmopts')
Note: "@"-file is overridden by local '.jvmopts' or '-jvm-opts' argument.
-jvm-opts <path> file containing jvm args (if not given, .jvmopts in project root is used if present)
-Dkey=val pass -Dkey=val directly to the jvm
-J-X pass option -X directly to the jvm (-J is stripped)
# passing options to sbt, OR to this runner
SBT_OPTS environment variable holding either the sbt args directly, or
the reference to a file containing sbt args if given path is prepended by '@' (e.g. '@/etc/sbtopts')
Note: "@"-file is overridden by local '.sbtopts' or '-sbt-opts' argument.
-sbt-opts <path> file containing sbt args (if not given, .sbtopts in project root is used if present)
-S-X add -X to sbt's scalacOptions (-S is stripped)
EOM
}
process_args () {
require_arg () {
local type="$1"
local opt="$2"
local arg="$3"
if [[ -z "$arg" ]] || [[ "${arg:0:1}" == "-" ]]; then
die "$opt requires <$type> argument"
fi
}
while [[ $# -gt 0 ]]; do
case "$1" in
-h|-help) usage; exit 1 ;;
-v) verbose=true && shift ;;
-d) addSbt "--debug" && addSbt debug && shift ;;
-w) addSbt "--warn" && addSbt warn && shift ;;
-q) addSbt "--error" && addSbt error && shift ;;
-x) debugUs=true && shift ;;
-trace) require_arg integer "$1" "$2" && trace_level="$2" && shift 2 ;;
-ivy) require_arg path "$1" "$2" && addJava "-Dsbt.ivy.home=$2" && shift 2 ;;
-no-colors) addJava "-Dsbt.log.noformat=true" && shift ;;
-no-share) noshare=true && shift ;;
-sbt-boot) require_arg path "$1" "$2" && addJava "-Dsbt.boot.directory=$2" && shift 2 ;;
-sbt-dir) require_arg path "$1" "$2" && sbt_dir="$2" && shift 2 ;;
-debug-inc) addJava "-Dxsbt.inc.debug=true" && shift ;;
-offline) addSbt "set offline := true" && shift ;;
-jvm-debug) require_arg port "$1" "$2" && addDebugger "$2" && shift 2 ;;
-batch) batch=true && shift ;;
-prompt) require_arg "expr" "$1" "$2" && setThisBuild shellPrompt "(s => { val e = Project.extract(s) ; $2 })" && shift 2 ;;
-sbt-create) sbt_create=true && shift ;;
-sbt-jar) require_arg path "$1" "$2" && sbt_jar="$2" && shift 2 ;;
-sbt-version) require_arg version "$1" "$2" && sbt_explicit_version="$2" && shift 2 ;;
-sbt-force-latest) sbt_explicit_version="$sbt_release_version" && shift ;;
-sbt-dev) sbt_explicit_version="$sbt_unreleased_version" && shift ;;
-sbt-launch-dir) require_arg path "$1" "$2" && sbt_launch_dir="$2" && shift 2 ;;
-sbt-launch-repo) require_arg path "$1" "$2" && sbt_launch_repo="$2" && shift 2 ;;
-scala-version) require_arg version "$1" "$2" && setScalaVersion "$2" && shift 2 ;;
-binary-version) require_arg version "$1" "$2" && setThisBuild scalaBinaryVersion "\"$2\"" && shift 2 ;;
-scala-home) require_arg path "$1" "$2" && setThisBuild scalaHome "scala.Some(file(\"$2\"))" && shift 2 ;;
-java-home) require_arg path "$1" "$2" && setJavaHome "$2" && shift 2 ;;
-sbt-opts) require_arg path "$1" "$2" && sbt_opts_file="$2" && shift 2 ;;
-jvm-opts) require_arg path "$1" "$2" && jvm_opts_file="$2" && shift 2 ;;
-D*) addJava "$1" && shift ;;
-J*) addJava "${1:2}" && shift ;;
-S*) addScalac "${1:2}" && shift ;;
-28) setScalaVersion "$latest_28" && shift ;;
-29) setScalaVersion "$latest_29" && shift ;;
-210) setScalaVersion "$latest_210" && shift ;;
-211) setScalaVersion "$latest_211" && shift ;;
-212) setScalaVersion "$latest_212" && shift ;;
--debug) addSbt debug && addResidual "$1" && shift ;;
--warn) addSbt warn && addResidual "$1" && shift ;;
--error) addSbt error && addResidual "$1" && shift ;;
*) addResidual "$1" && shift ;;
esac
done
}
# process the direct command line arguments
process_args "$@"
# skip #-styled comments and blank lines
readConfigFile() {
local end=false
until $end; do
read || end=true
[[ $REPLY =~ ^# ]] || [[ -z $REPLY ]] || echo "$REPLY"
done < "$1"
}
# if there are file/environment sbt_opts, process again so we
# can supply args to this runner
if [[ -r "$sbt_opts_file" ]]; then
vlog "Using sbt options defined in file $sbt_opts_file"
while read opt; do extra_sbt_opts+=("$opt"); done < <(readConfigFile "$sbt_opts_file")
elif [[ -n "$SBT_OPTS" && ! ("$SBT_OPTS" =~ ^@.*) ]]; then
vlog "Using sbt options defined in variable \$SBT_OPTS"
extra_sbt_opts=( $SBT_OPTS )
else
vlog "No extra sbt options have been defined"
fi
[[ -n "${extra_sbt_opts[*]}" ]] && process_args "${extra_sbt_opts[@]}"
# reset "$@" to the residual args
set -- "${residual_args[@]}"
argumentCount=$#
# set sbt version
set_sbt_version
# only exists in 0.12+
setTraceLevel() {
case "$sbt_version" in
"0.7."* | "0.10."* | "0.11."* ) echoerr "Cannot set trace level in sbt version $sbt_version" ;;
*) setThisBuild traceLevel $trace_level ;;
esac
}
# set scalacOptions if we were given any -S opts
[[ ${#scalac_args[@]} -eq 0 ]] || addSbt "set scalacOptions in ThisBuild += \"${scalac_args[@]}\""
# Update build.properties on disk to set explicit version - sbt gives us no choice
[[ -n "$sbt_explicit_version" ]] && update_build_props_sbt "$sbt_explicit_version"
vlog "Detected sbt version $sbt_version"
[[ -n "$scala_version" ]] && vlog "Overriding scala version to $scala_version"
# no args - alert them there's stuff in here
(( argumentCount > 0 )) || {
vlog "Starting $script_name: invoke with -help for other options"
residual_args=( shell )
}
# verify this is an sbt dir or -create was given
[[ -r ./build.sbt || -d ./project || -n "$sbt_create" ]] || {
cat <<EOM
$(pwd) doesn't appear to be an sbt project.
If you want to start sbt anyway, run:
$0 -sbt-create
EOM
exit 1
}
# pick up completion if present; todo
[[ -r .sbt_completion.sh ]] && source .sbt_completion.sh
# no jar? download it.
[[ -r "$sbt_jar" ]] || acquire_sbt_jar || {
# still no jar? uh-oh.
echo "Download failed. Obtain the jar manually and place it at $sbt_jar"
exit 1
}
if [[ -n "$noshare" ]]; then
for opt in ${noshare_opts}; do
addJava "$opt"
done
else
case "$sbt_version" in
"0.7."* | "0.10."* | "0.11."* | "0.12."* )
[[ -n "$sbt_dir" ]] || {
sbt_dir="$HOME/.sbt/$sbt_version"
vlog "Using $sbt_dir as sbt dir, -sbt-dir to override."
}
;;
esac
if [[ -n "$sbt_dir" ]]; then
addJava "-Dsbt.global.base=$sbt_dir"
fi
fi
if [[ -r "$jvm_opts_file" ]]; then
vlog "Using jvm options defined in file $jvm_opts_file"
while read opt; do extra_jvm_opts+=("$opt"); done < <(readConfigFile "$jvm_opts_file")
elif [[ -n "$JVM_OPTS" && ! ("$JVM_OPTS" =~ ^@.*) ]]; then
vlog "Using jvm options defined in \$JVM_OPTS variable"
extra_jvm_opts=( $JVM_OPTS )
else
vlog "Using default jvm options"
extra_jvm_opts=( $(default_jvm_opts) )
fi
# traceLevel is 0.12+
[[ -n "$trace_level" ]] && setTraceLevel
main () {
execRunner "$java_cmd" \
"${extra_jvm_opts[@]}" \
"${java_args[@]}" \
-jar "$sbt_jar" \
"${sbt_commands[@]}" \
"${residual_args[@]}"
}
# sbt inserts this string on certain lines when formatting is enabled:
# val OverwriteLine = "\r\u001BM\u001B[2K"
# ...in order not to spam the console with a million "Resolving" lines.
# Unfortunately that makes it that much harder to work with when
# we're not going to print those lines anyway. We strip that bit of
# line noise, but leave the other codes to preserve color.
mainFiltered () {
local ansiOverwrite='\r\x1BM\x1B[2K'
local excludeRegex=$(egrep -v '^#|^$' ~/.sbtignore | paste -sd'|' -)
echoLine () {
local line="$1"
local line1="$(echo "$line" | sed 's/\r\x1BM\x1B\[2K//g')" # This strips the OverwriteLine code.
local line2="$(echo "$line1" | sed 's/\x1B\[[0-9;]*[JKmsu]//g')" # This strips all codes - we test regexes against this.
if [[ $line2 =~ $excludeRegex ]]; then
[[ -n $debugUs ]] && echo "[X] $line1"
else
[[ -n $debugUs ]] && echo " $line1" || echo "$line1"
fi
}
echoLine "Starting sbt with output filtering enabled."
main | while read -r line; do echoLine "$line"; done
}
# Only filter if there's a filter file and we don't see a known interactive command.
# Obviously this is super ad hoc but I don't know how to improve on it. Testing whether
# stdin is a terminal is useless because most of my use cases for this filtering are
# exactly when I'm at a terminal, running sbt non-interactively.
shouldFilter () { [[ -f ~/.sbtignore ]] && ! egrep -q '\b(shell|console|consoleProject)\b' <<<"${residual_args[@]}"; }
# run sbt
if shouldFilter; then mainFiltered; else main; fi
================================================
FILE: src/main/scala/spinoco/fs2/http/HttpClient.scala
================================================
package spinoco.fs2.http
import java.nio.channels.AsynchronousChannelGroup
import java.util.concurrent.TimeUnit
import cats.Applicative
import javax.net.ssl.SSLContext
import cats.effect._
import fs2._
import fs2.concurrent.SignallingRef
import fs2.io.tcp.Socket
import scodec.{Codec, Decoder, Encoder}
import spinoco.fs2.http.internal.{addressForRequest, clientLiftToSecure, readWithTimeout}
import spinoco.fs2.http.sse.{SSEDecoder, SSEEncoding}
import spinoco.fs2.http.websocket.{Frame, WebSocket, WebSocketRequest}
import spinoco.protocol.http.header._
import spinoco.protocol.mime.MediaType
import spinoco.protocol.http.{HttpRequestHeader, HttpResponseHeader}
import scala.concurrent.ExecutionContext
import scala.concurrent.duration._
trait HttpClient[F[_]] {
/**
* Performs a single `request`. Returns one response if client replied.
*
* Note that request may contain stream of bytes that shall be sent to client.
* The response from server is evaluated _after_ client sent all data, including the body to the server.
*
* Note that the evaluation of `body` in HttpResponse may not outlive scope of resulting stream. That means
* only correct way to process the result is within the flatMap i.e.
* `
* request(thatRequest).flatMap { response =>
* response.body.through(bodyProcessor)
* }
* `
*
* This methods allows to be supplied with timeout (default is 5s) that the request awaits to be completed before
* failure.
*
* Timeout is computed once the requests was sent and includes also the time for processing the response header
* but not the body.
*
* Resulting stream fails with TimeoutException if the timeout is triggered
*
* @param request Request to make to server
* @param chunkSize Size of the chunk to used when receiving response from server
* @param timeout Request will fail if response header and response body is not received within supplied timeout
*
*/
def request(
request: HttpRequest[F]
, chunkSize: Int = 32*1024
, maxResponseHeaderSize: Int = 4096
, timeout: Duration = 5.seconds
):Stream[F,HttpResponse[F]]
/**
* Establishes websocket connection to the server.
*
* Implementation is according to RFC-6455 (https://tools.ietf.org/html/rfc6455).
*
* If this is established successfully, then this consults `pipe` to receive/sent any frames
* From/To server. Once the connection finishes, this will emit once None.
*
* If the connection was not established correctly (i.e. Authorization failure) this will not
* consult supplied pipe and instead this will immediately emit response received from the server.
*
* @param request WebSocket request
* @param pipe Pipe that is consulted when WebSocket is established correctly
* @param maxResponseHeaderSize Max size of Http Response header received
* @param chunkSize Size of receive buffer to use
* @param maxFrameSize Maximum size of single WebSocket frame. If the binary size of single frame is larger than
* supplied value, WebSocket will fail.
*
*/
def websocket[I : Decoder, O : Encoder](
request: WebSocketRequest
, pipe: Pipe[F, Frame[I], Frame[O]]
, maxResponseHeaderSize: Int = 4096
, chunkSize: Int = 32 * 1024
, maxFrameSize: Int = 1024*1024
): Stream[F, Option[HttpResponseHeader]]
/**
* Reads SSE encoded stream of data from the server.
*
* @param request Request to server. Note that this must be `GET` request.
* @param maxResponseHeaderSize Max size of expected response header
* @param chunkSize Max size of the chunk
*/
def sse[A : SSEDecoder](
request: HttpRequest[F]
, maxResponseHeaderSize: Int = 4096
, chunkSize: Int = 32 * 1024
): Stream[F, A]
}
object HttpClient {
/**
* Creates an Http Client
* @param requestCodec Codec used to decode request header
* @param responseCodec Codec used to encode response header
* @param sslExecutionContext Strategy used when communication with SSL (https or wss)
* @param sslContext SSL Context to use with SSL Client (https, wss)
*/
def apply[F[_] : ConcurrentEffect : ContextShift : Timer](
requestCodec : Codec[HttpRequestHeader]
, responseCodec : Codec[HttpResponseHeader]
, sslExecutionContext: => ExecutionContext
, sslContext : => SSLContext
)(implicit AG: AsynchronousChannelGroup):F[HttpClient[F]] = Sync[F].delay {
lazy val sslCtx = sslContext
lazy val sslS = sslExecutionContext
new HttpClient[F] {
def request(
request: HttpRequest[F]
, chunkSize: Int
, maxResponseHeaderSize: Int
, timeout: Duration
): Stream[F, HttpResponse[F]] = {
Stream.eval(addressForRequest[F](request.scheme, request.host)).flatMap { address =>
Stream.resource(io.tcp.client[F](address))
.evalMap { socket =>
if (!request.isSecure) Applicative[F].pure(socket)
else clientLiftToSecure[F](sslS, sslCtx)(socket, request.host)
}
.flatMap { impl.request[F](request, chunkSize, maxResponseHeaderSize, timeout, requestCodec, responseCodec ) }}
}
def websocket[I : Decoder, O : Encoder](
request: WebSocketRequest
, pipe: Pipe[F, Frame[I], Frame[O]]
, maxResponseHeaderSize: Int
, chunkSize: Int
, maxFrameSize: Int
): Stream[F, Option[HttpResponseHeader]] =
WebSocket.client(request,pipe,maxResponseHeaderSize,chunkSize,maxFrameSize, requestCodec, responseCodec, sslS, sslCtx)
def sse[A : SSEDecoder](rq: HttpRequest[F], maxResponseHeaderSize: Int, chunkSize: Int): Stream[F, A] =
request(rq, chunkSize, maxResponseHeaderSize, Duration.Inf).flatMap { resp =>
if (resp.header.headers.exists {
case `Content-Type`(ct) => ct.mediaType == MediaType.`text/event-stream`
case _ => false
})
resp.body through SSEEncoding.decodeA[F, A]
else
Stream.raiseError(new Throwable(s"Received response is not SSE: $resp"))
}
}
}
private[http] object impl {
def request[F[_] : Concurrent](
request: HttpRequest[F]
, chunkSize: Int
, maxResponseHeaderSize: Int
, timeout: Duration
, requestCodec: Codec[HttpRequestHeader]
, responseCodec: Codec[HttpResponseHeader]
)(socket: Socket[F])(implicit clock: Clock[F]):Stream[F, HttpResponse[F]] = {
import Stream._
timeout match {
case fin: FiniteDuration =>
eval(clock.realTime(TimeUnit.MILLISECONDS)).flatMap { start =>
HttpRequest.toStream(request, requestCodec).to(socket.writes(Some(fin))).last.onFinalize(socket.endOfOutput).flatMap { _ =>
eval(SignallingRef[F, Boolean](true)).flatMap { timeoutSignal =>
eval(clock.realTime(TimeUnit.MILLISECONDS)).flatMap { sent =>
val remains = fin - (sent - start).millis
readWithTimeout(socket, remains, timeoutSignal.get, chunkSize)
.through (HttpResponse.fromStream[F](maxResponseHeaderSize, responseCodec))
.flatMap { response =>
eval_(timeoutSignal.set(false)) ++ emit(response)
}
}}}}
case _ =>
HttpRequest.toStream(request, requestCodec).to(socket.writes(None)).last.onFinalize(socket.endOfOutput).flatMap { _ =>
socket.reads(chunkSize, None) through HttpResponse.fromStream[F](maxResponseHeaderSize, responseCodec)
}
}
}
}
}
================================================
FILE: src/main/scala/spinoco/fs2/http/HttpRequestOrResponse.scala
================================================
package spinoco.fs2.http
import cats.effect.Sync
import fs2.Chunk.ByteVectorChunk
import fs2.{Stream, _}
import scodec.Attempt.{Failure, Successful}
import scodec.{Attempt, Codec, Err}
import spinoco.fs2.http.body.{BodyDecoder, BodyEncoder, StreamBodyEncoder}
import spinoco.protocol.http._
import header._
import spinoco.protocol.mime.{ContentType, MediaType}
import scodec.bits.ByteVector
import spinoco.fs2.http.sse.{SSEEncoder, SSEEncoding}
/** common request/response methods **/
sealed trait HttpRequestOrResponse[F[_]] { self =>
type Self <: HttpRequestOrResponse[F]
/** yields to true, if body of this request shall be chunked **/
lazy val bodyIsChunked : Boolean =
withHeaders(internal.bodyIsChunked)
/** allows to stream arbitrary sized stream of `A` to remote party (i.e. upload) **/
def withStreamBody[A](body: Stream[F, A])(implicit E: StreamBodyEncoder[F, A]): Self = {
updateBody(body through E.encode)
.withContentType(E.contentType)
.asInstanceOf[Self]
}
/** like `stream` except one `A` that is streamed lazily **/
def withStreamBody1[A](a: => A)(implicit E: StreamBodyEncoder[F, A]): Self =
withStreamBody(Stream.suspend(Stream.emit(a)))
/** sets body size to supplied value **/
def withBodySize(sz: Long): Self =
updateHeaders(withHeaders(internal.swapHeader(`Content-Length`(sz))))
/** gets body size, if one specified **/
def bodySize: Option[Long] =
withHeaders(_.collectFirst { case `Content-Length`(sz) => sz })
protected def body: Stream[F, Byte]
/** encodes body `A` given BodyEncoder exists **/
def withBody[A](a: A)(implicit W: BodyEncoder[A], ev: RaiseThrowable[F]): Self = {
W.encode(a) match {
case Failure(err) => updateBody(body = Stream.raiseError(new Throwable(s"failed to encode $a: $err")))
case Successful(bytes) =>
val headers = withHeaders {
_.filterNot { h => h.isInstanceOf[`Content-Type`] || h.isInstanceOf[`Content-Length`] } ++
List(`Content-Type`(W.contentType), `Content-Length`(bytes.size))
}
updateBody(Stream.chunk(ByteVectorChunk(bytes)))
.updateHeaders(headers)
.asInstanceOf[Self]
}
}
/** encodes body as utf8 string **/
def withUtf8Body(s: String)(implicit ev: RaiseThrowable[F]): Self =
withBody(s)(BodyEncoder.utf8String, ev)
/** Decodes body with supplied decoder of `A` **/
def bodyAs[A](implicit D: BodyDecoder[A], F: Sync[F]): F[Attempt[A]] = {
withHeaders { _.collectFirst { case `Content-Type`(ct) => ct } match {
case None => F.pure(Attempt.failure(Err("Content type is not known")))
case Some(ct) =>
F.map(self.body.chunks.map(util.chunk2ByteVector).compile.toVector) { bs =>
if (bs.isEmpty) Attempt.failure(Err("Body is empty"))
else D.decode(bs.reduce(_ ++ _), ct)
}
}}
}
/** gets body as stream of byteVectors **/
def bodyAsByteVectorStream:Stream[F,ByteVector] =
self.body.chunks.map(util.chunk2ByteVector)
/** decodes body as string with encoding supplied in ContentType **/
def bodyAsString(implicit F: Sync[F]): F[Attempt[String]] =
bodyAs[String](BodyDecoder.stringDecoder, F)
/** updates content type to one specified **/
def withContentType(ct: ContentType): Self =
updateHeaders(withHeaders(internal.swapHeader(`Content-Type`(ct))))
/** gets ContentType, if one specififed **/
def contentType: Option[ContentType] =
withHeaders(_.collectFirst{ case `Content-Type`(ct) => ct })
/** configures encoding as chunked **/
def chunkedEncoding: Self =
updateHeaders(withHeaders(internal.swapHeader(`Transfer-Encoding`(List("chunked")))))
def withHeaders[A](f: List[HttpHeader] => A): A = self match {
case HttpRequest(_,_,header,_) => f(header.headers)
case HttpResponse(header, _) => f(header.headers)
}
/** appends supplied headers **/
def appendHeader(header: HttpHeader, headers: HttpHeader*): Self =
updateHeaders(withHeaders(_ ++ (header +: headers.toSeq)))
/** appends supplied headers. Unlike `appendHeader` headers are removed if they already exists **/
def withHeader(header : HttpHeader, headers: HttpHeader*): Self =
updateHeaders(withHeaders { current =>
val allNew = header +: headers
val allNewKeys = allNew.map(_.name.toLowerCase).toSet
current.filterNot(h => allNewKeys.contains(h.name.toLowerCase)) ++ allNew
})
protected def updateBody(body: Stream[F, Byte]): Self
protected def updateHeaders(headers: List[HttpHeader]): Self
}
/**
* Model of Http Request sent by client.
*
* @param host Host/port where to perform the request to
* @param header Header of the request
* @param body Body of the request. If empty, no body will be emitted.
*/
final case class HttpRequest[F[_]](
scheme: Scheme
, host: HostPort
, header: HttpRequestHeader
, body: Stream[F, Byte]
) extends HttpRequestOrResponse[F] { self =>
type Self = HttpRequest[F]
def withMethod(method: HttpMethod.Value): HttpRequest[F] = {
self.copy(header = self.header.copy(method = method))
}
def isSecure: Boolean = scheme match {
case HttpScheme.HTTPS | HttpScheme.WSS => true
case _ => false
}
protected def updateBody(body: Stream[F, Byte]): Self =
self.copy(body = body)
protected def updateHeaders(headers: List[HttpHeader]): Self =
self.copy(header = self.header.copy(headers = headers))
/**
* Encodes query params to body as `application/x-www-form-urlencoded` content.
* That means instead of passing query as part of request, they are encoded as utf8 body.
* @return
*/
def withQueryBodyEncoded(q:Uri.Query)(implicit ev: RaiseThrowable[F]): Self =
withBody(q)(BodyEncoder.`x-www-form-urlencoded`, ev)
def bodyAsQuery(implicit F: Sync[F]):F[Attempt[Uri.Query]] =
bodyAs[Uri.Query](BodyDecoder.`x-www-form-urlencoded`, F)
/**
* Adds supplied query as param in the Uri
*/
def withQuery(query:Uri.Query): Self =
self.copy(header = self.header.copy( query = query ))
}
object HttpRequest {
def get[F[_]](uri: Uri): HttpRequest[F] =
HttpRequest(
scheme = uri.scheme
, host = uri.host
, header = HttpRequestHeader(
method = HttpMethod.GET
, path = uri.path
, query = uri.query
, headers = List(
Host(uri.host)
)
)
, body = Stream.empty)
def post[F[_] : RaiseThrowable, A](uri: Uri, a: A)(implicit E: BodyEncoder[A]): HttpRequest[F] =
get(uri).withMethod(HttpMethod.POST).withBody(a)
def put[F[_] : RaiseThrowable, A](uri: Uri, a: A)(implicit E: BodyEncoder[A]): HttpRequest[F] =
get(uri).withMethod(HttpMethod.PUT).withBody(a)
def delete[F[_]](uri: Uri): HttpRequest[F] =
get(uri).withMethod(HttpMethod.DELETE)
/**
* Reads http header and body from the stream of bytes.
*
* If the body is encoded in chunked encoding this will decode it
*
* @param maxHeaderSize Maximum size of the http header
* @param headerCodec header codec to use
* @tparam F
* @return
*/
def fromStream[F[_] : RaiseThrowable](
maxHeaderSize: Int
, headerCodec: Codec[HttpRequestHeader]
): Pipe[F, Byte, (HttpRequestHeader, Stream[F, Byte])] = {
import internal._
_ through httpHeaderAndBody(maxHeaderSize) flatMap { case (header, bodyRaw) =>
headerCodec.decodeValue(header.bits) match {
case Failure(err) => Stream.raiseError(new Throwable(s"Decoding of the request header failed: $err"))
case Successful(decoded) =>
val body =
if (bodyIsChunked(decoded.headers)) bodyRaw through ChunkedEncoding.decode(1000)
else bodyRaw
Stream.emit(decoded -> body)
}
}
}
/**
* Converts the supplied request to binary stream of data to be sent over wire.
* Note that this inspects the headers to eventually perform chunked encoding of the stream,
* if that indication is present in headers,
*
* Otherwise this just encodes as binary stream of data after header of the request.
*
*
* @param request request to convert to stream
* @param headerCodec Codec to convert the header to bytes
*/
def toStream[F[_] : RaiseThrowable](
request: HttpRequest[F]
, headerCodec: Codec[HttpRequestHeader]
): Stream[F, Byte] = Stream.suspend {
import internal._
headerCodec.encode(request.header) match {
case Failure(err) => Stream.raiseError(new Throwable(s"Encoding of the header failed: $err"))
case Successful(bits) =>
val body =
if (request.bodyIsChunked) request.body through ChunkedEncoding.encode
else request.body
Stream.chunk[F, Byte](ByteVectorChunk(bits.bytes ++ `\r\n\r\n`)) ++ body
}
}
}
/**
* Model of Http Response
*
* @param header Header of the response
* @param body Body of the response. If empty, no body will be emitted.
*/
final case class HttpResponse[F[_]](
header: HttpResponseHeader
, body: Stream[F, Byte]
) extends HttpRequestOrResponse[F] { self =>
override type Self = HttpResponse[F]
protected def updateBody(body: Stream[F, Byte]): Self =
self.copy(body = body)
protected def updateHeaders(headers: List[HttpHeader]): Self =
self.copy(header= self.header.copy(headers = headers))
/** encodes supplied stream of `A` as SSE stream in body **/
def sseBody[A](in: Stream[F, A])(implicit E: SSEEncoder[A], ev: RaiseThrowable[F]): Self =
self
.updateBody(in through SSEEncoding.encodeA[F, A])
.updateHeaders(withHeaders(internal.swapHeader(`Content-Type`(ContentType.TextContent(MediaType.`text/event-stream`, None)))))
}
object HttpResponse {
def apply[F[_]](sc: HttpStatusCode):HttpResponse[F] = {
HttpResponse(
header = HttpResponseHeader(status = sc, reason = sc.label)
, body = Stream.empty
)
}
/**
* Decodes stream of bytes as HttpResponse.
*/
def fromStream[F[_] : RaiseThrowable](
maxHeaderSize: Int
, responseCodec: Codec[HttpResponseHeader]
): Pipe[F,Byte, HttpResponse[F]] = {
import internal._
_ through httpHeaderAndBody(maxHeaderSize) flatMap { case (header, bodyRaw) =>
responseCodec.decodeValue(header.bits) match {
case Failure(err) => Stream.raiseError(new Throwable(s"Failed to decode http response :$err"))
case Successful(response) =>
val unboundedBody =
if (bodyIsChunked(response.headers)) bodyRaw through ChunkedEncoding.decode(1024)
else bodyRaw
val contentLengthOpt = response.headers collectFirst {
case `Content-Length`(value) => value
}
val body = contentLengthOpt.fold(unboundedBody)(unboundedBody.take)
Stream.emit(HttpResponse(response, body))
}
}
}
/** Encodes response to stream of bytes **/
def toStream[F[_] : RaiseThrowable](
response: HttpResponse[F]
, headerCodec: Codec[HttpResponseHeader]
): Stream[F, Byte] = Stream.suspend {
import internal._
headerCodec.encode(response.header) match {
case Failure(err) => Stream.raiseError(new Throwable(s"Failed to encode http response : $response :$err "))
case Successful(encoded) =>
val body =
if (bodyIsChunked(response.header.headers)) response.body through ChunkedEncoding.encode
else response.body
Stream.chunk[F, Byte](ByteVectorChunk(encoded.bytes ++ `\r\n\r\n`)) ++ body
}
}
}
================================================
FILE: src/main/scala/spinoco/fs2/http/HttpServer.scala
================================================
package spinoco.fs2.http
import java.net.InetSocketAddress
import java.nio.channels.AsynchronousChannelGroup
import cats.effect.{ConcurrentEffect, Sync, Timer}
import cats.syntax.all._
import fs2._
import fs2.concurrent.SignallingRef
import scodec.Codec
import spinoco.protocol.http.codec.{HttpRequestHeaderCodec, HttpResponseHeaderCodec}
import spinoco.protocol.http.{HttpRequestHeader, HttpResponseHeader, HttpStatusCode}
import scala.concurrent.duration._
object HttpServer {
/**
* Creates simple http server,
*
* Serve will run after the resulting stream is run.
*
* @param bindTo Address and port where to bind server to
* @param maxConcurrent Maximum requests to process concurrently
* @param receiveBufferSize Receive buffer size for each connection
* @param maxHeaderSize Maximum size of http header for incoming requests, in bytes
* @param requestHeaderReceiveTimeout A timeout to await request header to be fully received.
* Request will fail, if the header won't be read within this timeout.
* @param requestCodec Codec for Http Request Header
* @param service Pipe that defines handling of each incoming request and produces a response
* @param requestFailure A function to be evaluated when server failed to read the request header.
* This may generate the default server response on unexpected failure.
* This is also evaluated when the server failed to process the request itself (i.e. `service` did not handle the failure )
* @param sendFailure A function to be evaluated on failure to process the the response.
* Request is not suplied if failure happened before request was constructed.
*
*/
def apply[F[_] : ConcurrentEffect : Timer](
maxConcurrent: Int = Int.MaxValue
, receiveBufferSize: Int = 256 * 1024
, maxHeaderSize: Int = 10 *1024
, requestHeaderReceiveTimeout: Duration = 5.seconds
, requestCodec: Codec[HttpRequestHeader] = HttpRequestHeaderCodec.defaultCodec
, responseCodec: Codec[HttpResponseHeader] = HttpResponseHeaderCodec.defaultCodec
, bindTo: InetSocketAddress
, service: (HttpRequestHeader, Stream[F,Byte]) => Stream[F,HttpResponse[F]]
, requestFailure : Throwable => Stream[F, HttpResponse[F]]
, sendFailure: (Option[HttpRequestHeader], HttpResponse[F], Throwable) => Stream[F, Nothing]
)(
implicit
AG: AsynchronousChannelGroup
): Stream[F, Unit] = {
import Stream._
import internal._
val (initial, readDuration) = requestHeaderReceiveTimeout match {
case fin: FiniteDuration => (true, fin)
case _ => (false, 0.millis)
}
io.tcp.server[F](bindTo, receiveBufferSize = receiveBufferSize).map { resource =>
Stream.resource(resource).flatMap { socket =>
eval(SignallingRef(initial)).flatMap { timeoutSignal =>
readWithTimeout[F](socket, readDuration, timeoutSignal.get, receiveBufferSize)
.through(HttpRequest.fromStream(maxHeaderSize, requestCodec))
.flatMap { case (request, body) =>
eval_(timeoutSignal.set(false)) ++
service(request, body).take(1).handleErrorWith { rsn => requestFailure(rsn).take(1) }
.map { resp => (request, resp) }
}
.attempt
.evalMap { attempt =>
def send(request:Option[HttpRequestHeader], resp: HttpResponse[F]): F[Unit] = {
HttpResponse.toStream(resp, responseCodec).through(socket.writes()).onFinalize(socket.endOfOutput).compile.drain.attempt flatMap {
case Left(err) => sendFailure(request, resp, err).compile.drain
case Right(()) => Sync[F].pure(())
}
}
attempt match {
case Right((request, response)) => send(Some(request), response)
case Left(err) => requestFailure(err).evalMap { send(None, _) }.compile.drain
}
}
.drain
}
}}.parJoin(maxConcurrent)
}
/** default handler for parsing request errors **/
def handleRequestParseError[F[_] : RaiseThrowable](err: Throwable): Stream[F, HttpResponse[F]] = {
Stream.suspend {
err.printStackTrace()
Stream.emit(HttpResponse[F](HttpStatusCode.BadRequest))
}.covary[F]
}
/** default handler for failures of sending request/response **/
def handleSendFailure[F[_]](header: Option[HttpRequestHeader], response: HttpResponse[F], err:Throwable): Stream[F, Nothing] = {
Stream.suspend {
err.printStackTrace()
Stream.empty
}
}
}
================================================
FILE: src/main/scala/spinoco/fs2/http/body/BodyDecoder.scala
================================================
package spinoco.fs2.http.body
import scodec.bits.ByteVector
import scodec.{Attempt, Decoder, Err}
import spinoco.protocol.http.Uri
import spinoco.protocol.mime.{ContentType, MIMECharset, MediaType}
import spinoco.fs2.http.util
trait BodyDecoder[A] {
def decode(bytes: ByteVector, contentType: ContentType): Attempt[A]
}
object BodyDecoder {
@inline def apply[A](implicit instance: BodyDecoder[A]): BodyDecoder[A] = instance
def instance[A](f: (ByteVector, ContentType) => Attempt[A]): BodyDecoder[A] =
new BodyDecoder[A] {
def decode(bytes: ByteVector, contentType: ContentType): Attempt[A] =
f(bytes, contentType)
}
def forDecoder[A](f: ContentType => Attempt[Decoder[A]]): BodyDecoder[A] =
BodyDecoder.instance { (bs, ct) => f(ct).flatMap(_.decodeValue(bs.bits)) }
val stringDecoder: BodyDecoder[String] = BodyDecoder.instance { case (bytes, ct) =>
if (! ct.mediaType.isText) Attempt.Failure(Err(s"Media Type must be text, but is ${ct.mediaType}"))
else {
MIMECharset.asJavaCharset(util.getCharset(ct).getOrElse(MIMECharset.`UTF-8`)).flatMap { implicit chs =>
Attempt.fromEither(bytes.decodeString.left.map(ex => Err(s"Failed to decode string ContentType: $ct, charset: $chs, err: ${ex.getMessage}")))
}
}
}
/** decodes body as query encoded as application/x-www-form-urlencoded data **/
val `x-www-form-urlencoded`: BodyDecoder[Uri.Query] =
forDecoder { ct =>
if (ct.mediaType == MediaType.`application/x-www-form-urlencoded`) Attempt.successful(Uri.Query.codec)
else Attempt.failure(Err(s"Unsupported content type : $ct"))
}
}
================================================
FILE: src/main/scala/spinoco/fs2/http/body/BodyEncoder.scala
================================================
package spinoco.fs2.http.body
import scodec.bits.ByteVector
import scodec.{Attempt, Encoder, Err}
import spinoco.protocol.http.Uri
import spinoco.protocol.mime.{ContentType, MIMECharset, MediaType}
/**
* Encodes one `A` to body, strictly
*/
sealed trait BodyEncoder[A] { self =>
def encode(a: A): Attempt[ByteVector]
def contentType: ContentType
/** given f, converts to encoder BodyEncoder[F, B] **/
def mapIn[B](f: B => A): BodyEncoder[B] =
BodyEncoder.instance(self.contentType) { b => self.encode(f(b)) }
/** given f, converts to encoder BodyEncoder[F, B] **/
def mapInAttempt[B](f: B => Attempt[A]): BodyEncoder[B] =
BodyEncoder.instance(self.contentType) { b => f(b).flatMap(self.encode) }
def withContentType(tpe: ContentType): BodyEncoder[A] =
BodyEncoder.instance(tpe)(self.encode)
}
object BodyEncoder {
@inline def apply[A](implicit instance: BodyEncoder[A]): BodyEncoder[A] = instance
def instance[A](tpe: ContentType)(f: A => Attempt[ByteVector]): BodyEncoder[A] =
new BodyEncoder[A] {
def encode(a: A): Attempt[ByteVector] = f(a)
def contentType: ContentType = tpe
}
def byteVector(tpe: ContentType = ContentType.BinaryContent(MediaType.`application/octet-stream`, None)): BodyEncoder[ByteVector] =
BodyEncoder.instance(tpe)(Attempt.successful)
val utf8String: BodyEncoder[String] =
BodyEncoder.instance(ContentType.TextContent(MediaType.`text/plain`, Some(MIMECharset.`UTF-8`))){ s =>
Attempt.fromEither(ByteVector.encodeUtf8(s).left.map { ex => Err(s"Failed to encode string: ${ex.getMessage}, ($s)") })
}
def forEncoder[A](tpe: ContentType)(codec: Encoder[A]):BodyEncoder[A] =
BodyEncoder.instance(tpe)(a => codec.encode(a).map(_.bytes))
/** encodes supplied query as application/x-www-form-urlencoded data **/
def `x-www-form-urlencoded`: BodyEncoder[Uri.Query] =
forEncoder(ContentType.TextContent(MediaType.`application/x-www-form-urlencoded`, None))(Uri.Query.codec)
}
================================================
FILE: src/main/scala/spinoco/fs2/http/body/StreamBodyDecoder.scala
================================================
package spinoco.fs2.http.body
import fs2._
import spinoco.protocol.mime.{ContentType, MIMECharset}
import spinoco.fs2.http.util
sealed trait StreamBodyDecoder[F[_], A] {
/** decodes stream with supplied content type. yields to None, if the ContentType is not of required type **/
def decode(ct: ContentType): Option[Pipe[F, Byte, A]]
}
object StreamBodyDecoder {
@inline def apply[F[_], A](implicit instance: StreamBodyDecoder[F, A]): StreamBodyDecoder[F, A] = instance
def instance[F[_], A](f: ContentType => Option[Pipe[F, Byte, A]]): StreamBodyDecoder[F, A] =
new StreamBodyDecoder[F, A] { def decode(ct: ContentType): Option[Pipe[F, Byte, A]] = f(ct) }
def utf8StringDecoder[F[_]]: StreamBodyDecoder[F, String] =
StreamBodyDecoder.instance { ct =>
if (ct.mediaType.isText && util.getCharset(ct).contains(MIMECharset.`UTF-8`)) Some(text.utf8Decode[F])
else None
}
}
================================================
FILE: src/main/scala/spinoco/fs2/http/body/StreamBodyEncoder.scala
================================================
package spinoco.fs2.http.body
import cats.MonadError
import fs2.Chunk.ByteVectorChunk
import fs2._
import scodec.Attempt.{Failure, Successful}
import scodec.bits.ByteVector
import spinoco.protocol.mime.{ContentType, MIMECharset, MediaType}
trait StreamBodyEncoder[F[_], A] {
/** an pipe to encode stram of `A` to stream of bytes **/
def encode: Pipe[F, A, Byte]
def contentType: ContentType
/** given f, converts to encoder BodyEncoder[F, B] **/
def mapIn[B](f: B => A): StreamBodyEncoder[F, B] =
StreamBodyEncoder.instance(contentType) { _ map f through encode }
/** given f, converts to encoder BodyEncoder[F, B] **/
def mapInF[B](f: B => F[A]): StreamBodyEncoder[F, B] =
StreamBodyEncoder.instance(contentType) { _ evalMap f through encode }
/** changes content type of this encoder **/
def withContentType(tpe: ContentType): StreamBodyEncoder[F, A] =
StreamBodyEncoder.instance(tpe)(encode)
}
object StreamBodyEncoder {
@inline def apply[F[_], A](implicit instance: StreamBodyEncoder[F, A]): StreamBodyEncoder[F, A] = instance
def instance[F[_], A](tpe: ContentType)(pipe: Pipe[F, A, Byte]): StreamBodyEncoder[F, A] =
new StreamBodyEncoder[F, A] {
def contentType: ContentType = tpe
def encode: Pipe[F, A, Byte] = pipe
}
/** encoder that encodes bytes as they come in, with `application/octet-stream` content type **/
def byteEncoder[F[_]] : StreamBodyEncoder[F, Byte] =
StreamBodyEncoder.instance(ContentType.BinaryContent(MediaType.`application/octet-stream`, None)) { identity }
/** encoder that encodes ByteVector as they come in, with `application/octet-stream` content type **/
def byteVectorEncoder[F[_]] : StreamBodyEncoder[F, ByteVector] =
StreamBodyEncoder.instance(ContentType.BinaryContent(MediaType.`application/octet-stream`, None)) { _.flatMap { bv => Stream.chunk(ByteVectorChunk(bv)) } }
/** encoder that encodes utf8 string, with `text/plain` utf8 content type **/
def utf8StringEncoder[F[_]](implicit F: MonadError[F, Throwable]) : StreamBodyEncoder[F, String] =
byteVectorEncoder mapInF[String] { s =>
ByteVector.encodeUtf8(s) match {
case Right(bv) => F.pure(bv)
case Left(err) => F.raiseError[ByteVector](new Throwable(s"Failed to encode string: $err ($s) "))
}
} withContentType ContentType.TextContent(MediaType.`text/plain`, Some(MIMECharset.`UTF-8`))
/** a convenience wrapper to convert body encoder to StreamBodyEncoder **/
def fromBodyEncoder[F[_] : RaiseThrowable, A](implicit E: BodyEncoder[A]):StreamBodyEncoder[F, A] =
StreamBodyEncoder.instance(E.contentType) { _.flatMap { a =>
E.encode(a) match {
case Failure(err) => Stream.raiseError(new Throwable(s"Failed to encode: $err ($a)"))
case Successful(bytes) => Stream.chunk(ByteVectorChunk(bytes))
}
}}
}
================================================
FILE: src/main/scala/spinoco/fs2/http/http.scala
================================================
package spinoco.fs2
import java.net.InetSocketAddress
import java.nio.channels.AsynchronousChannelGroup
import java.util.concurrent.Executors
import javax.net.ssl.SSLContext
import cats.effect.{ConcurrentEffect, ContextShift, Timer}
import fs2._
import scodec.Codec
import spinoco.protocol.http.{HttpRequestHeader, HttpResponseHeader}
import spinoco.protocol.http.codec.{HttpRequestHeaderCodec, HttpResponseHeaderCodec}
import scala.concurrent.ExecutionContext
import scala.concurrent.duration._
package object http {
/**
* Creates simple http server,
*
* Serve will run after the resulting stream is run.
*
* @param bindTo Address and port where to bind server to
* @param maxConcurrent Maximum requests to process concurrently
* @param receiveBufferSize Receive buffer size for each connection
* @param maxHeaderSize Maximum size of http header for incoming requests, in bytes
* @param requestHeaderReceiveTimeout A timeout to await request header to be fully received.
* Request will fail, if the header won't be read within this timeout.
* @param service Pipe that defines handling of each incoming request and produces a response
*/
def server[F[_] : ConcurrentEffect : Timer](
bindTo: InetSocketAddress
, maxConcurrent: Int = Int.MaxValue
, receiveBufferSize: Int = 256 * 1024
, maxHeaderSize: Int = 10 *1024
, requestHeaderReceiveTimeout: Duration = 5.seconds
, requestCodec: Codec[HttpRequestHeader] = HttpRequestHeaderCodec.defaultCodec
, responseCodec: Codec[HttpResponseHeader] = HttpResponseHeaderCodec.defaultCodec
)(
service: (HttpRequestHeader, Stream[F,Byte]) => Stream[F,HttpResponse[F]]
)(implicit AG: AsynchronousChannelGroup):Stream[F,Unit] = HttpServer(
maxConcurrent = maxConcurrent
, receiveBufferSize = receiveBufferSize
, maxHeaderSize = maxHeaderSize
, requestHeaderReceiveTimeout = requestHeaderReceiveTimeout
, requestCodec = requestCodec
, responseCodec = responseCodec
, bindTo = bindTo
, service = service
, requestFailure = HttpServer.handleRequestParseError[F] _
, sendFailure = HttpServer.handleSendFailure[F] _
)
/**
* Creates a client that can be used to make http requests to servers
*
* @param requestCodec Codec used to decode request header
* @param responseCodec Codec used to encode response header
* @param sslStrategy Strategy used to perform blocking SSL operations
*/
def client[F[_]: ConcurrentEffect : ContextShift : Timer](
requestCodec: Codec[HttpRequestHeader] = HttpRequestHeaderCodec.defaultCodec
, responseCodec: Codec[HttpResponseHeader] = HttpResponseHeaderCodec.defaultCodec
, sslStrategy: => ExecutionContext = ExecutionContext.fromExecutorService(Executors.newCachedThreadPool(util.mkThreadFactory("fs2-http-ssl", daemon = true)))
, sslContext: => SSLContext = { val ctx = SSLContext.getInstance("TLS"); ctx.init(null,null,null); ctx }
)(implicit AG: AsynchronousChannelGroup):F[HttpClient[F]] =
HttpClient(requestCodec, responseCodec, sslStrategy, sslContext)
}
================================================
FILE: src/main/scala/spinoco/fs2/http/internal/ChunkedEncoding.scala
================================================
package spinoco.fs2.http.internal
import fs2.Chunk.ByteVectorChunk
import fs2._
import scodec.bits.ByteVector
import spinoco.fs2.http.util.chunk2ByteVector
/**
* Created by pach on 20/01/17.
*/
object ChunkedEncoding {
/** decodes from the HTTP chunked encoding. After last chunk this terminates. Allows to specify max header size, after which this terminates
* Please see https://en.wikipedia.org/wiki/Chunked_transfer_encoding for details
*/
def decode[F[_] : RaiseThrowable](maxChunkHeaderSize:Int): Pipe[F, Byte, Byte] = {
// on left reading the header of chunk (acting as buffer)
// on right reading the chunk itself, and storing remaining bytes of the chunk
def go(expect:Either[ByteVector,Long], in: Stream[F, Byte]): Pull[F, Byte, Unit] = {
in.pull.uncons.flatMap {
case None => Pull.done
case Some((h, tl)) =>
val bv = chunk2ByteVector(h)
expect match {
case Left(header) =>
val nh = header ++ bv
val endOfheader = nh.indexOfSlice(`\r\n`)
if (endOfheader == 0) go(expect, Stream.chunk(ByteVectorChunk(bv.drop(`\r\n`.size))) ++ tl) //strip any leading crlf on header, as this starts with /r/n
else if (endOfheader < 0 && nh.size > maxChunkHeaderSize) Pull.raiseError(new Throwable(s"Failed to get Chunk header. Size exceeds max($maxChunkHeaderSize) : ${nh.size} ${nh.decodeUtf8}"))
else if (endOfheader < 0) go(Left(nh), tl)
else {
val (hdr,rem) = nh.splitAt(endOfheader + `\r\n`.size)
readChunkedHeader(hdr.dropRight(`\r\n`.size)) match {
case None => Pull.raiseError(new Throwable(s"Failed to parse chunked header : ${hdr.decodeUtf8}"))
case Some(0) => Pull.done
case Some(sz) => go(Right(sz), Stream.chunk(ByteVectorChunk(rem)) ++ tl)
}
}
case Right(remains) =>
if (remains == bv.size) Pull.output(ByteVectorChunk(bv)) >> go(Left(ByteVector.empty), tl)
else if (remains > bv.size) Pull.output(ByteVectorChunk(bv)) >> go(Right(remains - bv.size), tl)
else {
val (out,next) = bv.splitAt(remains.toInt)
Pull.output(ByteVectorChunk(out)) >> go(Left(ByteVector.empty), Stream.chunk(ByteVectorChunk(next)) ++ tl)
}
}
}
}
go(Left(ByteVector.empty), _) stream
}
private val lastChunk: Chunk[Byte] = ByteVectorChunk((ByteVector('0') ++ `\r\n` ++ `\r\n`).compact)
/**
* Encodes chunk of bytes to http chunked encoding.
*/
def encode[F[_]]:Pipe[F,Byte,Byte] = {
def encodeChunk(bv:ByteVector):Chunk[Byte] = {
if (bv.isEmpty) Chunk.empty
else ByteVectorChunk(ByteVector.view(bv.size.toHexString.toUpperCase.getBytes) ++ `\r\n` ++ bv ++ `\r\n` )
}
_.mapChunks { ch => encodeChunk(chunk2ByteVector(ch)) } ++ Stream.chunk(lastChunk)
}
/** yields to size of header in case the chunked header was succesfully parsed, else yields to None **/
private def readChunkedHeader(hdr:ByteVector):Option[Long] = {
hdr.decodeUtf8.right.toOption.flatMap { s =>
val parts = s.split(';') // lets ignore any extensions
if (parts.isEmpty) None
else {
try { Some(java.lang.Long.parseLong(parts(0).trim,16))}
catch { case t: Throwable => None }
}
}
}
}
================================================
FILE: src/main/scala/spinoco/fs2/http/internal/internal.scala
================================================
package spinoco.fs2.http
import java.net.InetSocketAddress
import java.util.concurrent.TimeoutException
import javax.net.ssl.{SNIHostName, SNIServerName, SSLContext}
import cats.effect.{Concurrent, ContextShift, Sync}
import cats.syntax.all._
import fs2.Chunk.ByteVectorChunk
import fs2.Stream._
import fs2.io.tcp.Socket
import fs2.{Stream, _}
import scodec.bits.ByteVector
import spinoco.fs2.crypto.io.tcp.TLSSocket
import spinoco.protocol.http.{HostPort, HttpScheme, Scheme}
import spinoco.protocol.http.header.{HttpHeader, `Transfer-Encoding`}
import scala.concurrent.ExecutionContext
import scala.concurrent.duration._
import scala.reflect.ClassTag
package object internal {
val `\n` : ByteVector = ByteVector('\n')
val `\r` : ByteVector = ByteVector('\r')
val `\r\n`: ByteVector = ByteVector('\r','\n')
val `\r\n\r\n` = (`\r\n` ++ `\r\n`).compact
/** yields to true, if chunked encoding header is present **/
def bodyIsChunked(headers:List[HttpHeader]):Boolean = {
headers.exists {
case `Transfer-Encoding`(encodings) => encodings.exists(_.equalsIgnoreCase("chunked"))
case _ => false
}
}
/**
* From the stream of bytes this extracts Http Header and body part.
*/
def httpHeaderAndBody[F[_] : RaiseThrowable](maxHeaderSize: Int): Pipe[F, Byte, (ByteVector, Stream[F, Byte])] = {
def go(buff: ByteVector, in: Stream[F, Byte]): Pull[F, (ByteVector, Stream[F, Byte]), Unit] = {
in.pull.uncons flatMap {
case None =>
Pull.raiseError(new Throwable(s"Incomplete Header received (sz = ${buff.size}): ${buff.decodeUtf8}"))
case Some((chunk, tl)) =>
val bv = spinoco.fs2.http.util.chunk2ByteVector(chunk)
val all = buff ++ bv
val idx = all.indexOfSlice(`\r\n\r\n`)
if (idx < 0) {
if (all.size > maxHeaderSize) Pull.raiseError(new Throwable(s"Size of the header exceeded the limit of $maxHeaderSize (${all.size})"))
else go(all, tl)
}
else {
val (h, t) = all.splitAt(idx)
if (h.size > maxHeaderSize) Pull.raiseError(new Throwable(s"Size of the header exceeded the limit of $maxHeaderSize (${all.size})"))
else Pull.output1((h, Stream.chunk(ByteVectorChunk(t.drop(`\r\n\r\n`.size))) ++ tl))
}
}
}
src => go(ByteVector.empty, src) stream
}
/** evaluates address from the host port and scheme, if this is a custom scheme we will default to port 8080**/
def addressForRequest[F[_] : Sync](scheme: Scheme, host: HostPort):F[InetSocketAddress] = Sync[F].delay {
val port = host.port.getOrElse {
scheme match {
case HttpScheme.HTTPS | HttpScheme.WSS => 443
case HttpScheme.HTTP | HttpScheme.WS => 80
case _ => 8080
}
}
new InetSocketAddress(host.host, port)
}
/** swaps header `H` for new value. If header exists, it is discarded. Appends header to the end**/
def swapHeader[H <: HttpHeader](header: H)(headers: List[HttpHeader])(implicit CT: ClassTag[H]) : List[HttpHeader] = {
headers.filterNot(CT.runtimeClass.isInstance) :+ header
}
/**
* Reads from supplied socket with timeout until `shallTimeout` yields to true.
* @param socket A socket to read from
* @param timeout A timeout
* @param shallTimeout If true, timeout will be applied, if false timeout won't be applied.
* @param chunkSize Size of chunk to read up to
*/
def readWithTimeout[F[_] : Sync](
socket: Socket[F]
, timeout: FiniteDuration
, shallTimeout: F[Boolean]
, chunkSize: Int
) : Stream[F, Byte] = {
def go(remains:FiniteDuration) : Stream[F, Byte] = {
eval(shallTimeout).flatMap { shallTimeout =>
if (!shallTimeout) socket.reads(chunkSize, None)
else {
if (remains <= 0.millis) Stream.raiseError(new TimeoutException())
else {
eval(Sync[F].delay(System.currentTimeMillis())).flatMap { start =>
eval(socket.read(chunkSize, Some(remains))).flatMap { read =>
eval(Sync[F].delay(System.currentTimeMillis())).flatMap { end => read match {
case Some(bytes) => Stream.chunk(bytes) ++ go(remains - (end - start).millis)
case None => Stream.empty
}}}}
}
}
}
}
go(timeout)
}
/** creates a function that lifts supplied socket to secure socket **/
def clientLiftToSecure[F[_] : Concurrent : ContextShift](sslES: => ExecutionContext, sslContext: => SSLContext)(socket: Socket[F], server: HostPort): F[Socket[F]] = {
import collection.JavaConverters._
Sync[F].delay {
val engine = sslContext.createSSLEngine(server.host, server.port.getOrElse(443))
val sslParams = engine.getSSLParameters
sslParams.setServerNames(List[SNIServerName](new SNIHostName(server.host)).asJava)
engine.setSSLParameters(sslParams)
engine.setUseClientMode(true)
engine
} flatMap {
TLSSocket.instance(socket, _, sslES)
.map(identity) //This is here just to make scala understand types properly
}
}
}
================================================
FILE: src/main/scala/spinoco/fs2/http/routing/MatchResult.scala
================================================
package spinoco.fs2.http.routing
import spinoco.fs2.http.HttpResponse
import spinoco.protocol.http.HttpStatusCode
trait MatchResult[+F[_],+A] { self =>
import MatchResult._
def map[B](f: A => B): MatchResult[F, B] = self match {
case Success(a) => Success(f(a))
case fail@Failed(_) => fail
}
def isSuccess: Boolean = self match {
case Success(_) => true
case _ => false
}
def isFailure: Boolean = ! isSuccess
def covary[F0[_] >: F[_]]: MatchResult[F0, A] = self.asInstanceOf[MatchResult[F0, A]]
}
object MatchResult {
implicit class MatchResultInvariantSyntax[F[_], A](val self: MatchResult[F,A]) extends AnyVal {
def fold[B](fa: HttpResponse[F] => B, fb: A => B):B = self match {
case Success(a) => fb(a)
case Failed(resp) => fa(resp.asInstanceOf[HttpResponse[F]])
}
}
case class Success[A](result: A) extends MatchResult[Nothing, A]
case class Failed[F[_]](response: HttpResponse[F]) extends MatchResult[F, Nothing]
def success[A](a: A) : MatchResult[Nothing, A] = Success(a)
def reply(code: HttpStatusCode):MatchResult[Nothing,Nothing] =
Failed[Nothing](HttpResponse[Nothing](code))
val NotFoundResponse: MatchResult[Nothing,Nothing] = reply(HttpStatusCode.NotFound)
val MethodNotAllowed: MatchResult[Nothing, Nothing] = reply(HttpStatusCode.MethodNotAllowed)
val BadRequest: MatchResult[Nothing, Nothing] = reply(HttpStatusCode.BadRequest)
}
================================================
FILE: src/main/scala/spinoco/fs2/http/routing/Matcher.scala
================================================
package spinoco.fs2.http.routing
import cats.effect.Sync
import fs2._
import shapeless.ops.function.FnToProduct
import shapeless.ops.hlist.Prepend
import shapeless.{::, HList, HNil}
import spinoco.fs2.http.HttpResponse
import spinoco.fs2.http.routing.MatchResult.{Failed, Success}
import spinoco.protocol.http.{HttpRequestHeader, HttpStatusCode, Uri}
sealed trait Matcher[+F[_], +A] { self =>
import Matcher._
/** transforms this matcher with supplied `f` **/
def map[B](f: A => B): Matcher[F, B] =
Bind[F, A, B](self, r => Matcher.ofResult(r.map(f)) )
/** defined ad map { _ => b} **/
def *>[B](b: B): Matcher[F, B] =
self.map { _ => b }
/** advances path by one segment, after this matches **/
def advance: Matcher[F, A] =
Advance(self)
/** matches this or yields to None **/
def ? : Matcher[F, Option[A]] =
self.map(Some(_)) or Matcher.success(None: Option[A])
}
object Matcher {
implicit class PureMatcherOps[F[_], A](val self: Matcher[F, A]) extends AnyVal {
/** like `map` but allows to evaluate `F` **/
def evalMap[B](f: A => F[B]): Matcher[F, B] =
self.flatMap { a => Eval(f(a)) }
/** transforms this matcher to another matcher with supplied `f` **/
def flatMap[B](f: A => Matcher[F, B]): Matcher[F, B] =
Bind[F, A, B](self, {
case success:Success[A] => f(success.result)
case failed:Failed[F] => Matcher.respond[F](failed.response)
})
/** allias for flatMap **/
def >>=[B](f: A => Matcher[F, B]): Matcher[F, B] =
flatMap(f)
/** defined as flatMap { _ => fb } **/
def >>[B](fb: Matcher[F, B]): Matcher[F, B] =
flatMap(_ => fb)
/** defined as flatMap { a => fb map { _ => a} } **/
def <<[B](fb: Matcher[F, B]): Matcher[F, A] =
flatMap(a => fb map { _ => a})
/** defined as advance.flatMap(f) **/
def />>=[B](f: A => Matcher[F, B]): Matcher[F, B] =
self.advance.flatMap(f)
/** like flatMap, but allows to apply `f` when match failed **/
def flatMapR[B](f: MatchResult[F,A] => Matcher[F, B]): Matcher[F, B] =
Bind[F, A, B](self, f)
/** applies `f` only when matcher fails to match **/
def recover(f: HttpResponse[F] => Matcher[F, A]): Matcher[F, A] =
Bind[F, A, A](self.asInstanceOf[Matcher[F, A]], {
case success: Success[A] => Matcher.success(success.result)
case failed: Failed[F] => f(failed.response)
})
/** matches and consumes current path segment throwing away `A` **/
def /[B](other : Matcher[F, B]): Matcher[F, B] =
self.advance.flatMap { _ => other }
/** matches and consumes current path segment throwing away `B` **/
def </[B](other: Matcher[F, B]): Matcher[F, A] =
self.advance.flatMap { a => other.map { _ => a } }
/** matches this or alternative **/
def or[A0 >: A](alt : => Matcher[F, A0]): Matcher[F, A0] =
Bind[F, A, A0](self, {
case success: Success[A] => Matcher.ofResult(success.asInstanceOf[MatchResult.Success[A0]])
case failed: Failed[F] => alt
})
def covary[F0[_] >: F[_]]:Matcher[F0, A] = self.asInstanceOf[Matcher[F0, A]]
}
// implicit class MatcherStringPathSyntax[F[_]](val self: Matcher[F, String]) extends AnyVal {
// def /(s: String): Matcher[F, String] =
// self.advance.flatMap { _ => uriSegment(s) }
//
// def or(s: String): Matcher[F, String] =
// Bind[F, String, String](self, {
// case success: Success[String] => Matcher.ofResult(success.asInstanceOf[MatchResult.Success[String]])
// case failed: Failed[F] => uriSegment(s)
// })
// } //TODO why is this here?
case class Match[F[_], A](f:(HttpRequestHeader, Stream[F, Byte]) => MatchResult[F, A]) extends Matcher[F, A]
case class Bind[F[_], A, B](m: Matcher[F, A], f: MatchResult[F,A] => Matcher[F, B]) extends Matcher[F, B]
case class Advance[F[_], A](m: Matcher[F, A]) extends Matcher[F, A]
case class Eval[F[_], A](f: F[A]) extends Matcher[F, A]
/** matcher that always succeeds **/
def success[A](a: A): Matcher[Nothing, A] =
Match[Nothing,A] { (_,_) => MatchResult.Success[A](a) }
/** matcher that always responds (fails) with supplied response **/
def respond[F[_]](response: HttpResponse[F]): Matcher[F, Nothing] =
Match[F, Nothing] { (_, _) => MatchResult.Failed[F](response) }
/** matcher that always responds with supplied status code **/
def respondWith(code: HttpStatusCode): Matcher[Nothing, Nothing] =
respond(HttpResponse(code))
/** Matcher that always results in result supplied**/
def ofResult[F[_], A](result:MatchResult[F,A]): Matcher[F, A] =
Match[F, A] { (_, _) => result }
/**
* Interprets matcher to obtain the result.
*/
def run[F[_], A](matcher: Matcher[F, A])(header: HttpRequestHeader, body: Stream[F, Byte])(implicit F: Sync[F]): F[MatchResult[F, A]] = {
def go[B](current:Matcher[F,B], path: Uri.Path):F[(MatchResult[F, B], Uri.Path)] = {
current match {
case m: Match[F,B] => F.map(F.pure(m.f(header.copy(path = path), body))) { _ -> path }
case m: Eval[F, B] => F.map(m.f)(b => Success(b) -> path)
case m: Bind[F, _, B] => F.flatMap(F.suspend(go(m.m, path))){ case (r, path0) =>
if (r.isSuccess) go(m.f(r), path0)
else go(m.f(r), path)
}
case m: Advance[F, B] => F.map(F.suspend(go(m.m, path))){ case (r, path0) =>
if (r.isSuccess) {
if (path0.segments.nonEmpty) r -> path0.copy(segments = path0.segments.tail)
else if (path0.trailingSlash) r -> path0.copy(trailingSlash = false)
else r -> path0 // no op
}
else r -> path
}
}
}
F.map(go(matcher, header.path)) { _._1 }
}
implicit class RequestMatcherHListSyntax[F[_], L <: HList](val self: Matcher[F, L]) extends AnyVal {
/** combines two matcher'r result to resulting hlist **/
def ::[B](other: Matcher[F, B]): Matcher[F, B :: L] =
other.flatMap { b => self.map { l => b :: l } }
/** combines this matcher with other matcher appending result of other matcher at the end **/
def :+[B](other: Matcher[F, B])(implicit P : Prepend[L, B :: HNil]): Matcher[F, P.Out] =
self.flatMap { l => other.map { b => l :+ b } }
/** prepends result of other matcher before the result of this matcher **/
def :::[L2 <: HList, HL <: HList](other: Matcher[F, L2])(implicit P: Prepend.Aux[L2, L, HL]): Matcher[F, HL] =
other.flatMap { l2 => self.map { l => l2 ::: l } }
/** combines two matcher'r result to resulting hlist, and advances path between them **/
def :/:[B](other : Matcher[F, B]): Matcher[F, B :: L] =
other.advance.flatMap { b => self.map { l => b :: l } }
/** like `map` but instead (L:HList) => B, takes ordinary function **/
def mapH[FF, B](f: FF)(implicit F2P: FnToProduct.Aux[FF, L => B]): Matcher[F, B] =
self.map { l => F2P(f)(l) }
}
implicit class RequestMatcherSyntax[F[_], A](val self: Matcher[F, A]) extends AnyVal {
/** applies this matcher and if it is is successful then applies `other` returning result in HList B :: A :: HNil */
def :: [B](other : Matcher[F, B]): Matcher[F, B :: A :: HNil] =
other.flatMap { b => self.map { a => b :: a :: HNil } }
def :/:[B](other : Matcher[F, B]): Matcher[F, B :: A :: HNil] =
other.advance.flatMap { b => self.map { a => b :: a :: HNil } }
}
}
================================================
FILE: src/main/scala/spinoco/fs2/http/routing/StringDecoder.scala
================================================
package spinoco.fs2.http.routing
import scodec.bits.{Bases, ByteVector}
import shapeless.tag
import shapeless.tag.@@
import scala.reflect.ClassTag
import scala.util.Try
/**
* Decoder for `A` to be decoded from supplied String
*/
sealed trait StringDecoder[A] { self =>
/** decode `A` from supplied string **/
def decode(s:String): Option[A]
def map[B](f: A => B): StringDecoder[B] =
StringDecoder { s => self.decode(s).map(f) }
def mapO[B](f: A => Option[B]): StringDecoder[B] =
StringDecoder { s => self.decode(s).flatMap(f) }
def filter(f: A => Boolean): StringDecoder[A] =
StringDecoder { s => self.decode(s).filter(f) }
}
object StringDecoder {
def apply[A]( f: String => Option[A]):StringDecoder[A] =
new StringDecoder[A] { def decode(s: String): Option[A] = f(s) }
implicit val boolInstance: StringDecoder[Boolean] =
StringDecoder { s =>
if (s.equalsIgnoreCase("true") ) Some(true)
else if (s.equalsIgnoreCase("false")) Some(false)
else None
}
implicit val stringInstance: StringDecoder[String] =
StringDecoder { Some(_) }
implicit val byteInstance : StringDecoder[Byte] =
StringDecoder { s => Try { s.toByte }.toOption }
implicit val shortInstance : StringDecoder[Short] =
StringDecoder { s => Try { s.toShort }.toOption }
implicit val intInstance : StringDecoder[Int] =
StringDecoder { s => Try { s.toInt }.toOption }
implicit val longInstance : StringDecoder[Long] =
StringDecoder { s => Try { s.toLong }.toOption }
implicit val doubleInstance : StringDecoder[Double] =
StringDecoder { s => Try { s.toDouble }.toOption }
implicit val floatInstance : StringDecoder[Float] =
StringDecoder { s => Try { s.toFloat }.toOption }
implicit val bigIntInstance : StringDecoder[BigInt] =
StringDecoder { s => Try { BigInt(s) }.toOption }
implicit val bigDecimalInstance : StringDecoder[BigDecimal] =
StringDecoder { s => Try { BigDecimal(s) }.toOption }
implicit val base64UrlInstance: StringDecoder[ByteVector @@ Base64Url] =
StringDecoder { s => ByteVector.fromBase64(s,Bases.Alphabets.Base64Url).map(tag[Base64Url](_)) }
implicit def enumInstance[E <: Enumeration : ClassTag] : StringDecoder[E#Value] = {
val E = implicitly[ClassTag[E]].runtimeClass.getField("MODULE$").get((): Unit).asInstanceOf[Enumeration]
StringDecoder { s => Try { E.withName(s)}.toOption.map(_.asInstanceOf[E#Value]) }
}
}
================================================
FILE: src/main/scala/spinoco/fs2/http/routing/routing.scala
================================================
package spinoco.fs2.http
import cats.effect.{Concurrent, Effect, Timer}
import fs2._
import scodec.{Attempt, Decoder, Encoder}
import scodec.bits.Bases.Base64Alphabet
import scodec.bits.{Bases, ByteVector}
import shapeless.Typeable
import spinoco.fs2.http.body.{BodyDecoder, StreamBodyDecoder}
import spinoco.fs2.http.routing.MatchResult._
import spinoco.fs2.http.routing.Matcher.{Eval, Match}
import spinoco.protocol.http.header._
import spinoco.protocol.http.{HttpMethod, HttpRequestHeader, HttpStatusCode, Uri}
import spinoco.fs2.http.util.chunk2ByteVector
import spinoco.fs2.http.websocket.{Frame, WebSocket}
import scala.concurrent.duration._
package object routing {
type Route[F[_]] = Matcher[F, Stream[F, HttpResponse[F]]]
/** tags bytes encoded as Base64Url **/
sealed trait Base64Url
/** converts supplied route to function that is handled over to server to perform the routing **/
def route[F[_]](r:Route[F])(implicit F: Effect[F]):(HttpRequestHeader, Stream[F, Byte]) => Stream[F, HttpResponse[F]] = {
(header, body) =>
Stream.eval(Matcher.run[F, Stream[F, HttpResponse[F]]](r)(header, body)).flatMap { mr =>
mr.fold((resp : HttpResponse[F]) => Stream.emit(resp), identity )
}
}
implicit class StringMatcherSyntax(val self: String) extends AnyVal {
def /[F[_], A] (m: Matcher[F, A]) : Matcher[F, A] =
string2RequestMatcher(self) / m
def or[F[_]] (m: Matcher[F, String]) : Matcher[F, String] =
string2RequestMatcher(self) or m
}
implicit def string2RequestMatcher(s:String): Matcher[Nothing, String] =
as(StringDecoder.stringInstance.filter(_ == s))
/** matches supplied method **/
def method[F[_]](method: HttpMethod.Value): Matcher[F, HttpMethod.Value] =
Match[F, HttpMethod.Value] { (rq, body) =>
if (rq.method == method) MatchResult.Success(method)
else MatchResult.MethodNotAllowed
}
val Get = method(HttpMethod.GET)
val Put = method(HttpMethod.PUT)
val Post = method(HttpMethod.POST)
val Delete = method(HttpMethod.DELETE)
val Options = method(HttpMethod.OPTIONS)
/** matches to relative path in current context **/
def path: Matcher[Nothing, Uri.Path] = {
Match[Nothing, Uri.Path]{ (request, _) =>
Success[Uri.Path](request.path.copy(initialSlash = false))
}
}
/** matches any supplied matcher **/
def choice[F[_],A](matcher: Matcher[F, A], matchers: Matcher[F, A]*): Matcher[F, A] = {
def go(m: Matcher[F,A], next: Seq[Matcher[F, A]]): Matcher[F, A] = {
next.headOption match {
case None => m
case Some(nm) => m.flatMapR[A] {
case Success(a) => Matcher.success(a)
case f: Failed[F] => go(nm, next.tail)
}
}
}
go(matcher, matchers)
}
/** matches if remaining path segments are empty **/
val empty : Matcher[Nothing, Unit] =
Match[Nothing, Unit] { (request, _) =>
if (request.path.segments.isEmpty) Success(())
else NotFoundResponse
}
/** matches header of type `h` **/
def header[H <: HttpHeader](implicit T: Typeable[H]): Matcher[Nothing, H] =
Match[Nothing, H] { (request, _) =>
request.headers.collectFirst(Function.unlift(T.cast)) match {
case None => BadRequest
case Some(h) => Success(h)
}
}
/**
* Matches if query contains `key` and that can be decoded to `A` via supplied decoder
*/
def param[A](key: String)(implicit decoder: StringDecoder[A]) : Matcher[Nothing,A] =
Match[Nothing, A] { (header, _) =>
header.query.valueOf(key).flatMap(decoder.decode) match {
case None => BadRequest
case Some(a) => Success(a)
}
}
/** Decodes Base64 (Url) encoded binary data in parameter specified by `key` **/
def paramBase64(key: String, alphabet: Base64Alphabet = Bases.Alphabets.Base64Url): Matcher[Nothing, ByteVector] =
param[String](key).flatMap { s =>
ByteVector.fromBase64(s, alphabet) match {
case None => Matcher.respondWith(HttpStatusCode.BadRequest)
case Some(bv) => Matcher.success(bv)
}
}
/** decodes head of the path to `A` givne supplied decoder from string **/
def as[A](implicit decoder: StringDecoder[A]): Matcher[Nothing, A] =
Match[Nothing, A] { (request, _) =>
request.path.segments.headOption.flatMap(decoder.decode) match {
case None => NotFoundResponse
case Some(a) => Success(a)
}
}
/**
* Creates a Matcher that when supplied a pipe will create the websocket connection.
* `I` is received from the client and `O` is sent to client.
* Decoder (for I) and Encoder (for O) must be supplied.
*
* @param pingInterval An interval for the Ping / Pong protocol.
* @param handshakeTimeout An timeout to await for handshake to be successfull. If the handshake is not completed
* within supplied period, connection is terminated.
* @param maxFrameSize Maximum size of single websocket frame. If the binary size of single frame is larger than
* supplied value, websocket will fail.
*/
def websocket[F[_] : Concurrent : Timer, I : Decoder, O : Encoder](
pingInterval: Duration = 30.seconds
, handshakeTimeout: FiniteDuration = 10.seconds
, maxFrameSize: Int = 1024*1024
): Match[Nothing, (Pipe[F, Frame[I], Frame[O]]) => Stream[F, HttpResponse[F]]] =
Match[Nothing, (Pipe[F, Frame[I], Frame[O]]) => Stream[F, HttpResponse[F]]] { (request, body) =>
Success(
WebSocket.server[F, I, O](_, pingInterval, handshakeTimeout, maxFrameSize)(request, body)
)
}
/**
* Evaluates `f` returning its result as successful matcher
*/
def eval[F[_],A](f: F[A]): Matcher[F, A] =
Eval(f)
/** extracts body of the request **/
def body[F[_]]: BodyHelper[F] = new BodyHelper[F] {}
trait BodyHelper[F[_]] {
/**
* extract body as raw bytes w/o checking its content type bytes.
* If `Content-Length` header is provided, then up to that much bytes is consumed from the body.
* Otherwise this prouces a stream of bytes that is terminated after clients signal EOF
*/
def bytes: Matcher[F, Stream[F, Byte]] =
header[`Content-Length`].?.flatMap { maybeSized =>
Match[F, Stream[F, Byte]] { (_, body) =>
MatchResult.success(maybeSized.map(_.value).fold(body) { sz => body.take(sz) })
}
}
/** extracts body as stream of `A` **/
def stream[A](implicit D: StreamBodyDecoder[F, A]): Matcher[F, Stream[F, A]] =
header[`Content-Type`].flatMap { ct =>
bytes.flatMap { s =>
D.decode(ct.value) match {
case None => Matcher.ofResult(BadRequest)
case Some(decode) => Matcher.success(s through decode)
}
}
}
/** extracts last element of the `body` or responds BadRequest if body can't be extracted **/
def as[A](implicit D: BodyDecoder[A], F: Effect[F]): Matcher[F, A] = {
header[`Content-Type`].flatMap { ct =>
bytes.flatMap { s => eval {
F.map(s.chunks.compile.toVector) { chunks =>
val bytes =
if (chunks.isEmpty) ByteVector.empty
else chunks.map(chunk2ByteVector).reduce(_ ++ _)
D.decode(bytes, ct.value)
}
}}.flatMap {
case Attempt.Successful(a) => Matcher.success(a)
case Attempt.Failure(err) => Matcher.ofResult(BadRequest)
}
}
}
}
}
================================================
FILE: src/main/scala/spinoco/fs2/http/sse/SSEDecoder.scala
================================================
package spinoco.fs2.http.sse
import scodec.Attempt
sealed trait SSEDecoder[A] { self =>
def decode(in: SSEMessage): Attempt[A]
def map[B](f: A => B): SSEDecoder[B] =
SSEDecoder.instance { a => self.decode(a).map(f) }
}
object SSEDecoder {
@inline def apply[A](implicit instance: SSEDecoder[A]): SSEDecoder[A] = instance
def instance[A](f: SSEMessage => Attempt[A]): SSEDecoder[A] =
new SSEDecoder[A] { def decode(in: SSEMessage): Attempt[A] = f(in) }
}
================================================
FILE: src/main/scala/spinoco/fs2/http/sse/SSEEncoder.scala
================================================
package spinoco.fs2.http.sse
import scodec.Attempt
sealed trait SSEEncoder[A] { self =>
def encode(a: A) : Attempt[SSEMessage]
def mapIn[B](f: B => A): SSEEncoder[B] =
SSEEncoder.instance { b => self.encode(f(b)) }
}
object SSEEncoder {
@inline def apply[A](implicit instance: SSEEncoder[A]): SSEEncoder[A] = instance
def instance[A](f: A => Attempt[SSEMessage]): SSEEncoder[A] =
new SSEEncoder[A] { def encode(a: A): Attempt[SSEMessage] = f(a) }
/** simple encoder of string messages **/
val stringEncoder: SSEEncoder[String] =
SSEEncoder.instance { s => Attempt.successful(SSEMessage.SSEData(Vector(s), None, None)) }
}
================================================
FILE: src/main/scala/spinoco/fs2/http/sse/SSEEncoding.scala
================================================
package spinoco.fs2.http.sse
import fs2.Chunk.ByteVectorChunk
import fs2._
import scodec.Attempt
import scodec.bits.ByteVector
import spinoco.fs2.http.util.chunk2ByteVector
import scala.util.Try
import scala.concurrent.duration._
object SSEEncoding {
/**
* Encodes supplied stream of (messageTag, messageContent) to SSE Stream
*/
def encode[F[_]]: Pipe[F, SSEMessage, Byte] = {
_.flatMap {
case SSEMessage.SSEData(data, event, id) =>
val eventBytes = event.map { s => s"event: $s" }.toSeq
val dataBytes = data.map { s => s"data: $s" }
val idBytes = id.map { s => s"id: $s" }.toSeq
Stream.chunk(ByteVectorChunk(ByteVector.view((
eventBytes ++ dataBytes ++ idBytes).mkString("", "\n", "\n\n").getBytes
)))
case SSEMessage.SSERetry(duration) =>
Stream.chunk(ByteVectorChunk(ByteVector.view(
s"retry: ${duration.toMillis}\n\n".getBytes
)))
}
}
/** encodes stream of `A` as SSE Stream **/
def encodeA[F[_] : RaiseThrowable, A](implicit E: SSEEncoder[A]): Pipe[F, A, Byte] = {
_ flatMap { a => E.encode(a) match {
case Attempt.Successful(msg) => Stream.emit(msg)
case Attempt.Failure(err) => Stream.raiseError(new Throwable(s"Failed to encode $a : $err"))
}} through encode
}
private val StartBom = ByteVector.fromValidHex("feff")
/**
* Decodes stream of bytes to SSE Messages
*/
def decode[F[_] : RaiseThrowable]: Pipe[F, Byte, SSEMessage] = {
// drops initial Byte Order Mark, if present
def dropInitial(buff:ByteVector): Pipe[F, Byte, Byte] = {
_.pull.uncons.flatMap {
case None => Pull.raiseError(new Throwable("SSE Socket did not contain any data"))
case Some((chunk, next)) =>
val all = buff ++ chunk2ByteVector(chunk)
if (all.size < 2) (next through dropInitial(all)).pull.echo
else {
if (all.startsWith(StartBom)) Pull.output(ByteVectorChunk(all.drop(2))) >> next.pull.echo
else Pull.output(ByteVectorChunk(all)) >> next.pull.echo
}
}.stream
}
// makes lines out of incoming bytes. Lines are utf-8 decoded
// separated by \r\n or \n or \r
def mkLines: Pipe[F, Byte, String] =
_ through text.utf8Decode[F] through text.lines[F]
// makes lines for single event
// removes all the comments and splits by empty lines
// outgoing vectors are guaranteed tobe nonEmpty
// note that this splits by empty lines.
// the last event is emitted only if it is terminated by empty line
def mkEvents: Pipe[F, String, Seq[String]] = {
def go(buff: Vector[String]): Stream[F, String] => Pull[F, Seq[String], Unit] = {
_.pull.uncons flatMap {
case None => Pull.done
case Some((lines, tl)) =>
val event = lines.toList.takeWhile(_.nonEmpty)
// size of event lines is NOT equal with size of lines only when there is nonEmpty line
if (event.size == lines.size) go(buff ++ event)(tl)
else Pull.output1(buff ++ event) >> go(Vector.empty)(Stream.chunk(lines.drop(event.size + 1)) ++ tl)
}
}
src => go(Vector.empty)(src.filter(! _.startsWith(":"))).stream
}
// constructs SSE Message
// if message contains "retry" separate retry event is emitted
// im message contains multiple "event" or "id" values, only last one is used.
def mkMessage: Pipe[F, Seq[String], SSEMessage] = {
_.flatMap { lines =>
val data =
lines.map { line =>
val idx = line.indexOf(':')
if (idx < 0) line -> ""
else {
val (tag, data) = line.splitAt(idx)
val dataNoColon = data.drop(1)
val dataOut = if (dataNoColon.startsWith(" ")) dataNoColon.drop(1) else dataNoColon
tag -> dataOut
}
}
val (mData, mEvent, mId, mRetry) =
data.foldLeft((Vector.empty[String], Option.empty[String], Option.empty[String], Option.empty[FiniteDuration])) {
case ((d, event, id, retry), next) => next match {
case ("data", v) => (d :+ v, event, id, retry)
case ("event", v) => (d, Some(v), id, retry)
case ("id", v) => (d, event, Some(v), retry)
case ("retry", v) => (d, event, Some(v), Try { v.trim.toInt.millis }.toOption)
case _ => (d, event, id, retry)
}
}
Stream.emit(SSEMessage.SSEData(mData, mEvent, mId))
.filter(m => m.data.nonEmpty || m.event.nonEmpty || m.id.nonEmpty) ++
Stream.emits(mRetry.toSeq.map(SSEMessage.SSERetry.apply))
}
}
_ through dropInitial(ByteVector.empty) through mkLines through mkEvents through mkMessage
}
/** decodes stream of sse messages to `A`, given supplied decoder **/
def decodeA[F[_] : RaiseThrowable, A](implicit D: SSEDecoder[A]): Pipe[F, Byte, A] = {
_ through decode flatMap { msg =>
D.decode(msg) match {
case Attempt.Successful(a) => Stream.emit(a)
case Attempt.Failure(err) => Stream.raiseError(new Throwable(s"Failed do decode: $msg : $err"))
}
}
}
}
================================================
FILE: src/main/scala/spinoco/fs2/http/sse/SSEMessage.scala
================================================
package spinoco.fs2.http.sse
import scala.concurrent.duration.FiniteDuration
/**
* SSE Message modeled after
* https://www.w3.org/TR/2011/WD-eventsource-20111020/
*/
sealed trait SSEMessage
object SSEMessage {
/**
* SSE Data received
* @param data Data fields, received in singel event
* @param event Name of the event, if one provided, empty otherwise
* @param id Id of the event if one provided, empty otherwise.
*/
case class SSEData(data: Seq[String], event: Option[String], id: Option[String]) extends SSEMessage
case class SSERetry(retryIN: FiniteDuration) extends SSEMessage
}
================================================
FILE: src/main/scala/spinoco/fs2/http/util/util.scala
================================================
package spinoco.fs2.http
import java.lang.Thread.UncaughtExceptionHandler
import java.util.concurrent.{Executors, ThreadFactory}
import java.util.concurrent.atomic.AtomicInteger
import fs2.Chunk.ByteVectorChunk
import fs2._
import scodec.bits.{BitVector, ByteVector}
import scodec.bits.Bases.{Alphabets, Base64Alphabet}
import spinoco.protocol.mime.{ContentType, MIMECharset}
import scala.concurrent.ExecutionContext
import scala.util.control.NonFatal
package object util {
/**
* Encodes bytes to base64 encoded bytes [[http://tools.ietf.org/html/rfc4648#section-5 RF4648 section 5]]
* Encoding is done lazily to support very large Base64 bodies i.e. email, attachments..)
* @param alphabet Alphabet to use
* @return
*/
def encodeBase64Raw[F[_]](alphabet:Base64Alphabet): Pipe[F, Byte, Byte] = {
def go(rem:ByteVector): Stream[F,Byte] => Pull[F, Byte, Unit] = {
_.pull.uncons flatMap {
case None =>
if (rem.size == 0) Pull.done
else Pull.output(ByteVectorChunk(ByteVector.view(rem.toBase64(alphabet).getBytes)))
case Some((chunk, tl)) =>
val n = rem ++ chunk2ByteVector(chunk)
if (n.size/3 > 0) {
val pad = n.size % 3
val enc = n.dropRight(pad)
val out = Array.ofDim[Byte]((enc.size.toInt / 3) * 4)
var pos = 0
enc.toBitVector.grouped(6) foreach { group =>
val idx = group.padTo(8).shiftRight(2, signExtension = false).toByteVector.head
out(pos) = alphabet.toChar(idx).toByte
pos = pos + 1
}
Pull.output(ByteVectorChunk(ByteVector.view(out))) >> go(n.takeRight(pad))(tl)
} else {
go(n)(tl)
}
}
}
src => go(ByteVector.empty)(src).stream
}
/** encodes base64 encoded stream [[http://tools.ietf.org/html/rfc4648#section-5 RF4648 section 5]]. Whitespaces are ignored **/
def encodeBase64Url[F[_]]:Pipe[F, Byte, Byte] =
encodeBase64Raw(Alphabets.Base64Url)
/** encodes base64 encoded stream [[http://tools.ietf.org/html/rfc4648#section-4 RF4648 section 4]] **/
def encodeBase64[F[_]]:Pipe[F, Byte, Byte] =
encodeBase64Raw[F](Alphabets.Base64)
/**
* Decodes base64 encoded stream with supplied alphabet. Whitespaces are ignored.
* Decoding is lazy to support very large Base64 bodies (i.e. email)
*/
def decodeBase64Raw[F[_] : RaiseThrowable](alphabet:Base64Alphabet):Pipe[F, Byte, Byte] = {
val Pad = alphabet.pad
def go(remAcc:BitVector): Stream[F, Byte] => Pull[F, Byte, Unit] = {
_.pull.uncons flatMap {
case None => Pull.done
case Some((chunk,tl)) =>
val bv = chunk2ByteVector(chunk)
var acc = remAcc
var idx = 0
var term = false
try {
bv.foreach { b =>
b.toChar match {
case c if alphabet.ignore(c) => // ignore no-op
case Pad => term = true
case c =>
if (!term) acc = acc ++ BitVector(alphabet.toIndex(c)).drop(2)
else {
throw new IllegalArgumentException(s"Unexpected character '$c' at index $idx after padding character; only '=' and whitespace characters allowed after first padding character")
}
}
idx = idx + 1
}
val aligned = (acc.size / 8) * 8
if (aligned <= 0 && !term) go(acc)(tl)
else {
val (out, rem) = acc.splitAt(aligned)
if (term) Pull.output(ByteVectorChunk(out.toByteVector))
else Pull.output(ByteVectorChunk(out.toByteVector)) >> go(rem)(tl)
}
} catch {
case e: IllegalArgumentException =>
Pull.raiseError(new Throwable(s"Invalid base 64 encoding at index $idx", e))
}
}
}
src => go(BitVector.empty)(src).stream
}
/** decodes base64 encoded stream [[http://tools.ietf.org/html/rfc4648#section-5 RF4648 section 5]]. Whitespaces are ignored **/
def decodeBase64Url[F[_] : RaiseThrowable]:Pipe[F, Byte, Byte] =
decodeBase64Raw(Alphabets.Base64Url)
/** decodes base64 encoded stream [[http://tools.ietf.org/html/rfc4648#section-4 RF4648 section 4]] **/
def decodeBase64[F[_] : RaiseThrowable]:Pipe[F, Byte, Byte] =
decodeBase64Raw(Alphabets.Base64)
/** converts chunk of bytes to ByteVector **/
def chunk2ByteVector(chunk: Chunk[Byte]):ByteVector = {
chunk match {
case bv: ByteVectorChunk => bv.toByteVector
case other =>
val bs = other.toBytes
ByteVector(bs.values, bs.offset, bs.size)
}
}
/** converts ByteVector to chunk **/
def byteVector2Chunk(bv: ByteVector): Chunk[Byte] = {
ByteVectorChunk(bv)
}
/** helper to create named daemon thread factories **/
def mkThreadFactory(name: String, daemon: Boolean, exitJvmOnFatalError: Boolean = true): ThreadFactory = {
new ThreadFactory {
val idx = new AtomicInteger(0)
val defaultFactory = Executors.defaultThreadFactory()
def newThread(r: Runnable): Thread = {
val t = defaultFactory.newThread(r)
t.setName(s"$name-${idx.incrementAndGet()}")
t.setDaemon(daemon)
t.setUncaughtExceptionHandler(new UncaughtExceptionHandler {
def uncaughtException(t: Thread, e: Throwable): Unit = {
ExecutionContext.defaultReporter(e)
if (exitJvmOnFatalError) {
e match {
case NonFatal(_) => ()
case fatal => System.exit(-1)
}
}
}
})
t
}
}
}
def getCharset(ct: ContentType): Option[MIMECharset] = {
ct match {
case ContentType.TextContent(_, maybeCharset) => maybeCharset
case _ => None
}
}
}
================================================
FILE: src/main/scala/spinoco/fs2/http/websocket/Frame.scala
================================================
package spinoco.fs2.http.websocket
sealed trait Frame[A] { self =>
def a: A
def isText: Boolean = self match {
case Frame.Text(_) => true
case _ => false
}
def isBinary = !isText
}
object Frame {
case class Binary[A](a: A) extends Frame[A]
case class Text[A](a: A) extends Frame[A]
}
================================================
FILE: src/main/scala/spinoco/fs2/http/websocket/WebSocket.scala
================================================
package spinoco.fs2.http.websocket
import java.nio.channels.AsynchronousChannelGroup
import java.util.concurrent.Executors
import cats.Applicative
import javax.net.ssl.SSLContext
import cats.effect.{Concurrent, ConcurrentEffect, ContextShift, Timer}
import fs2.Chunk.ByteVectorChunk
import fs2._
import fs2.concurrent.Queue
import scodec.Attempt.{Failure, Successful}
import scodec.bits.ByteVector
import scodec.{Codec, Decoder, Encoder}
import spinoco.fs2.http.HttpResponse
import spinoco.protocol.http.codec.{HttpRequestHeaderCodec, HttpResponseHeaderCodec}
import spinoco.protocol.http.header._
import spinoco.protocol.http._
import spinoco.protocol.http.header.value.ProductDescription
import spinoco.protocol.mime.{ContentType, MIMECharset, MediaType}
import spinoco.protocol.websocket.{OpCode, WebSocketFrame}
import spinoco.protocol.websocket.codec.WebSocketFrameCodec
import spinoco.fs2.http.util.chunk2ByteVector
import scala.concurrent.ExecutionContext
import scala.concurrent.duration._
import scala.util.Random
object WebSocket {
/**
* Creates a websocket to be used on server side.
*
* Implementation is according to RFC-6455 (https://tools.ietf.org/html/rfc6455).
*
* @param pipe A websocket pipe. `I` is received from the client and `O` is sent to client.
* Decoder (for I) and Encoder (for O) must be supplied.
* @param pingInterval An interval for the Ping / Pong protocol.
* @param handshakeTimeout An timeout to await for handshake to be successfull. If the handshake is not completed
* within supplied period, connection is terminated.
* @param maxFrameSize Maximum size of single websocket frame. If the binary size of single frame is larger than
* supplied value, websocket will fail.
* @tparam F
* @return
*/
def server[F[_] : Concurrent : Timer, I : Decoder, O : Encoder](
pipe: Pipe[F, Frame[I], Frame[O]]
, pingInterval: Duration = 30.seconds
, handshakeTimeout: FiniteDuration = 10.seconds
, maxFrameSize: Int = 1024*1024
)(header: HttpRequestHeader, input:Stream[F,Byte]): Stream[F,HttpResponse[F]] = {
Stream.emit(
impl.verifyHeaderRequest[F](header).right.map { key =>
val respHeader = impl.computeHandshakeResponse(header, key)
HttpResponse(respHeader, input through impl.webSocketOf(pipe, pingInterval, maxFrameSize, client2Server = false))
}.merge
)
}
/**
* Establishes websocket connection to the server.
*
* Implementation is according to RFC-6455 (https://tools.ietf.org/html/rfc6455).
*
* If this is established successfully, then this consults `pipe` to receive/sent any frames
* From/To server. Once the connection finishes, this will emit once None.
*
* If the connection was not established correctly (i.e. Authorization failure) this will not
* consult supplied pipe and instead this will immediately emit response received from the server.
*
* @param request WebSocket request
* @param pipe Pipe that is consulted when websocket is established correctly
* @param maxHeaderSize Max size of Http Response header received
* @param receiveBufferSize Size of receive buffer to use
* @param maxFrameSize Maximum size of single websocket frame. If the binary size of single frame is larger than
* supplied value, websocket will fail.
* @param requestCodec Codec to encode HttpRequests Header
* @param responseCodec Codec to decode HttpResponse Header
*
*/
def client[F[_] : ConcurrentEffect : ContextShift : Timer, I : Decoder, O : Encoder](
request: WebSocketRequest
, pipe: Pipe[F, Frame[I], Frame[O]]
, maxHeaderSize: Int = 4096
, receiveBufferSize: Int = 256 * 1024
, maxFrameSize: Int = 1024*1024
, requestCodec: Codec[HttpRequestHeader] = HttpRequestHeaderCodec.defaultCodec
, responseCodec: Codec[HttpResponseHeader] = HttpResponseHeaderCodec.defaultCodec
, sslES: => ExecutionContext = ExecutionContext.fromExecutorService(Executors.newCachedThreadPool(spinoco.fs2.http.util.mkThreadFactory("fs2-http-ssl", daemon = true)))
, sslContext: => SSLContext = { val ctx = SSLContext.getInstance("TLS"); ctx.init(null,null,null); ctx }
)(implicit AG: AsynchronousChannelGroup): Stream[F, Option[HttpResponseHeader]] = {
import spinoco.fs2.http.internal._
import Stream._
eval(addressForRequest[F](if (request.secure) HttpScheme.WSS else HttpScheme.WS, request.hostPort)).flatMap { address =>
Stream.resource(io.tcp.client[F](address, receiveBufferSize = receiveBufferSize))
.evalMap { socket => if (request.secure) clientLiftToSecure(sslES, sslContext)(socket, request.hostPort) else Applicative[F].pure(socket) }
.flatMap { socket =>
val (header, fingerprint) = impl.createRequestHeaders(request.header)
requestCodec.encode(header) match {
case Failure(err) => Stream.raiseError(new Throwable(s"Failed to encode websocket request: $err"))
case Successful(headerBits) =>
eval(socket.write(ByteVectorChunk(headerBits.bytes ++ `\r\n\r\n`))).flatMap { _ =>
socket.reads(receiveBufferSize) through httpHeaderAndBody(maxHeaderSize) flatMap { case (respHeaderBytes, body) =>
responseCodec.decodeValue(respHeaderBytes.bits) match {
case Failure(err) => raiseError(new Throwable(s"Failed to decode websocket response: $err"))
case Successful(responseHeader) =>
impl.validateResponse[F](header, responseHeader, fingerprint).flatMap {
case Some(resp) => emit(Some(resp))
case None => (body through impl.webSocketOf(pipe, Duration.Undefined, maxFrameSize, client2Server = true) through socket.writes(None)).drain ++ emit(None)
}
}
}
}
}
}}
}
object impl {
private sealed trait PingPong
private object PingPong {
object Ping extends PingPong
object Pong extends PingPong
}
/**
* Verifies validity of WebSocket header request (on server) and extracts WebSocket key
*/
def verifyHeaderRequest[F[_]](header: HttpRequestHeader): Either[HttpResponse[F], ByteVector] = {
def badRequest(s:String) = HttpResponse[F](
header = HttpResponseHeader(
status = HttpStatusCode.BadRequest
, reason = HttpStatusCode.BadRequest.label
, headers = List(
`Content-Type`(ContentType.TextContent(MediaType.`text/plain`, Some(MIMECharset.`UTF-8`)))
)
)
, body = Stream.chunk(ByteVectorChunk(ByteVector.view(s.getBytes)))
)
def version: Either[HttpResponse[F], Int] = header.headers.collectFirst {
case `Sec-WebSocket-Version`(13) => Right(13)
case `Sec-WebSocket-Version`(other) => Left(badRequest(s"Unsupported websocket version: $other"))
}.getOrElse(Left(badRequest("Missing Sec-WebSocket-Version header")))
def host: Either[HttpResponse[F], Unit] = header.headers.collectFirst {
case Host(_) => Right(())
}.getOrElse(Left(badRequest("Missing header `Host: hostname`")))
def upgrade: Either[HttpResponse[F], Unit] = header.headers.collectFirst {
case Upgrade(pds) if pds.exists { pd => pd.name.equalsIgnoreCase("websocket") && pd.comment.isEmpty } => Right(())
}.getOrElse(Left(badRequest("Missing header `Upgrade: websocket`")))
def connection: Either[HttpResponse[F], Unit] = header.headers.collectFirst {
case Connection(s) if s.exists(_.equalsIgnoreCase("Upgrade")) => Right(())
}.getOrElse(Left(badRequest("Missing header `Connection: upgrade`")))
def webSocketKey: Either[HttpResponse[F], ByteVector] = header.headers.collectFirst {
case `Sec-WebSocket-Key`(key) => Right(key)
}.getOrElse(Left(badRequest("Missing Sec-WebSocket-Key header")))
for {
_ <- version.right
_ <- host.right
_ <- upgrade.right
_ <- connection.right
key <- webSocketKey.right
} yield key
}
/** creates the handshake response to complete websocket handshake on server side **/
def computeHandshakeResponse(header: HttpRequestHeader, key: ByteVector): HttpResponseHeader = {
val fingerprint = computeFingerPrint(key)
val headers = header.headers.collect {
case h: `Sec-WebSocket-Protocol` => h
}
HttpResponseHeader(
status = HttpStatusCode.SwitchingProtocols
, reason = HttpStatusCode.SwitchingProtocols.label
, headers = List(
Upgrade(List(ProductDescription("websocket", None)))
, Connection(List("Upgrade"))
, `Sec-WebSocket-Accept`(fingerprint)
) ++ headers
)
}
/**
* Creates websocket of supplied pipe
*
* @param pingInterval If Finite, defines duration when keep-alive pings are sent to client
* If client won't respond with pong to 3x this internal, the websocket will be terminated
* by server.
* @param client2Server When true, this represent client -> server direction, when false this represents reverse direction
*/
def webSocketOf[F[_] : Concurrent : Timer, I : Decoder, O : Encoder](
pipe: Pipe[F, Frame[I], Frame[O]]
, pingInterval: Duration
, maxFrameSize: Int
, client2Server: Boolean
):Pipe[F, Byte, Byte] = { source: Stream[F, Byte] => Stream.suspend {
Stream.eval(Queue.unbounded[F, PingPong]).flatMap { pingPongQ =>
val metronome: Stream[F, Unit] = pingInterval match {
case fin: FiniteDuration => Stream.awakeEvery[F](fin).map { _ => () }
case inf => Stream.empty
}
val control = controlStream[F](pingPongQ.dequeue, metronome, maxUnanswered = 3, flag = client2Server)
source
.through(decodeWebSocketFrame[F](maxFrameSize, client2Server))
.through(webSocketFrame2Frame[F, I](pingPongQ))
.through(pipe)
.through(frame2WebSocketFrame[F, O](if (client2Server) Some(Random.nextInt()) else None))
.mergeHaltBoth(control)
.through(encodeWebSocketFrame[F](client2Server))
}
}}
/**
* Cuts necessary data for decoding the frame, done by partially decoding
* the frame
* Empty if the data couldn't be decoded yet
*
* @param in Current buffer that may contain full frame
*/
def cutFrame(in:ByteVector): Option[ByteVector] = {
val bits = in.bits
if (bits.size < 16) None // smallest frame is 16 bits
else {
val maskSize = if (bits(8)) 4 else 0
val sz = bits.drop(9).take(7).toInt(signed = false)
val maybeEnough =
if (sz < 126) {
// no extended payload size, sz bytes expected
Some(sz.toLong + 2)
} else if (sz == 126) {
// next 16 bits is payload size
if (bits.size < 32) None
else Some(bits.drop(16).take(16).toInt(signed = false).toLong + 4)
} else {
// next 64 bits is payload size
if (bits.size < 80) None
else Some(bits.drop(16).take(64).toLong(signed = false) + 10)
}
maybeEnough.flatMap { sz =>
val fullSize = sz + maskSize
if (in.size < fullSize) None
else Some(in.take(fullSize))
}
}
}
/**
* Decodes websocket frame.
*
* This will fail when the frame failed to be decoded or when frame is larger than
* supplied `maxFrameSize` parameter.
*
* @param maxFrameSize Maximum size of the frame, including its header.
*/
def decodeWebSocketFrame[F[_] : RaiseThrowable](maxFrameSize: Int , flag: Boolean): Pipe[F, Byte, WebSocketFrame] = {
// Returns list of raw frames and tail of
// the buffer. Tail of the buffer cant be empty
// (or non-empty if last one frame isn't finalized).
def cutFrames(data: ByteVector, acc: Vector[ByteVector] = Vector.empty): (Vector[ByteVector], ByteVector) = {
cutFrame(data) match {
case Some(frameData) => cutFrames(data.drop(frameData.size), acc :+ frameData)
case None => (acc, data)
}
}
def go(buff: ByteVector): Stream[F, Byte] => Pull[F, WebSocketFrame, Unit] = { h0 =>
if (buff.size > maxFrameSize) Pull.raiseError(new Throwable(s"Size of websocket frame exceeded max size: $maxFrameSize, current: ${buff.size}, $buff"))
else {
h0.pull.uncons flatMap {
case None => Pull.done // todo: is ok to silently ignore buffer remainder ?
case Some((chunk, tl)) =>
val data = buff ++ chunk2ByteVector(chunk)
cutFrames(data) match {
case (rawFrames, _) if rawFrames.isEmpty => go(data)(tl)
case (rawFrames, dataTail) =>
val pulls = rawFrames.map { data =>
WebSocketFrameCodec.codec.decodeValue(data.bits) match {
case Failure(err) => Pull.raiseError(new Throwable(s"Failed to decode websocket frame: $err, $data"))
case Successful(wsFrame) => Pull.output1(wsFrame)
}
}
// pulls nonempty
pulls.reduce(_ >> _) >> go(dataTail)(tl)
}
}
}
}
src => go(ByteVector.empty)(src).stream
}
/**
* Collects incoming frames. to produce and deserialize Frame[A].
*
* Also interprets WebSocket operations.
* - if Ping is received, supplied Queue is enqueued with true
* - if Pong is received, supplied Queue is enqueued with false
* - if Close is received, the WebSocket is terminated
* - if Continuation is received, the buffer of the frame is enqueued and later used to deserialize to `A`.
*
* @param pongQ Queue to notify about ping/pong frames.
*/
def webSocketFrame2Frame[F[_] : RaiseThrowable, A](pongQ: Queue[F, PingPong])(implicit R: Decoder[A]): Pipe[F, WebSocketFrame, Frame[A]] = {
def decode(from: Vector[WebSocketFrame]):Pull[F, Frame[A], A] = {
val bs = from.map(_.payload).reduce(_ ++ _)
R.decodeValue(bs.bits) match {
case Failure(err) => Pull.raiseError(new Throwable(s"Failed to decode value: $err, content: $bs"))
case Successful(a) => Pull.pure(a)
}
}
def go(buff:Vector[WebSocketFrame]): Stream[F, WebSocketFrame] => Pull[F, Frame[A], Unit] = {
_.pull.uncons1 flatMap {
case None => Pull.done // todo: is ok to ignore remainder in buffer ?
case Some((frame, tl)) =>
frame.opcode match {
case OpCode.Continuation => go(buff :+ frame)(tl)
case OpCode.Text => decode(buff :+ frame).flatMap { decoded => Pull.output1(Frame.Text(decoded)) >> go(Vector.empty)(tl) }
case OpCode.Binary => decode(buff :+ frame).flatMap { decoded => Pull.output1(Frame.Binary(decoded)) >> go(Vector.empty)(tl) }
case OpCode.Ping => Pull.eval(pongQ.enqueue1(PingPong.Ping)) >> go(buff)(tl)
case OpCode.Pong => Pull.eval(pongQ.enqueue1(PingPong.Pong)) >> go(buff)(tl)
case OpCode.Close => Pull.done
}
}
}
src => go(Vector.empty)(src).stream
}
/**
* Encodes received frome to WebSocketFrame.
* @param maskKey A funtion that allows to generate random masking key. Masking is applied at client -> server direction only.
*/
def frame2WebSocketFrame[F[_] : RaiseThrowable, A](maskKey: => Option[Int])(implicit W: Encoder[A]): Pipe[F, Frame[A], WebSocketFrame] = {
_.flatMap { frame =>
W.encode(frame.a) match {
case Failure(err) => Stream.raiseError(new Throwable(s"Failed to encode frame: $err (frame: $frame)"))
case Successful(payload) =>
val opCode = if (frame.isText) OpCode.Text else OpCode.Binary
Stream.emit(WebSocketFrame(fin = true, (false, false, false), opCode, payload.bytes, maskKey))
}
}
}
private val pingFrame = WebSocketFrame(fin = true, (false, false, false), OpCode.Ping, ByteVector.empty, None)
private val pongFrame = WebSocketFrame(fin = true, (false, false, false), OpCode.Pong, ByteVector.empty, None)
private val closeFrame = WebSocketFrame(fin = true, (false, false, false), OpCode.Close, ByteVector.empty, None)
/**
* Encodes incoming frames to wire format.
* @tparam F
* @return
*/
def encodeWebSocketFrame[F[_] : RaiseThrowable](flag: Boolean): Pipe[F, WebSocketFrame, Byte] = {
_.append(Stream.emit(closeFrame)).flatMap { wsf =>
WebSocketFrameCodec.codec.encode(wsf) match {
case Failure(err) => Stream.raiseError(new Throwable(s"Failed to encode websocket frame: $err (frame: $wsf)"))
case Successful(data) => Stream.chunk(ByteVectorChunk(data.bytes))
}
}
}
/**
* Creates control stream. When control stream terminates WebSocket will terminate too.
*
* This takes ping-pong stream, for each Ping, this responds with Pong.
* For each Pong received this zeroes number of pings sent.
*
* @param pingPongs Stream of ping pongs received
* @param metronome A metronome that emits time to send Ping
* @param maxUnanswered Max unanswered pings to await before the stream terminates.
* @tparam F
* @return
*/
def controlStream[F[_] : Concurrent](
pingPongs: Stream[F, PingPong]
, metronome: Stream[F, Unit]
, maxUnanswered: Int
, flag: Boolean
): Stream[F, WebSocketFrame] = {
(pingPongs either metronome)
.mapAccumulate(0) { case (pingsSent, in) => in match {
case Left(PingPong.Pong) => (0, Stream.empty)
case Left(PingPong.Ping) => (pingsSent, Stream.emit(pongFrame))
case Right(_) => (pingsSent + 1, Stream.emit(pingFrame))
}}
.flatMap { case (unconfirmed, out) =>
if (unconfirmed < 3) out
else Stream.raiseError(new Throwable(s"Maximum number of unconfirmed pings exceeded: $unconfirmed"))
}
}
val magic = ByteVector.view("258EAFA5-E914-47DA-95CA-C5AB0DC85B11".getBytes)
def computeFingerPrint(key: ByteVector): ByteVector =
(ByteVector.view(key.toBase64.getBytes) ++ magic).digest("SHA-1")
/**
* Augments header to be correct for Websocket request (adding Sec-WebSocket-Key header) and
* returnng the correct header with expected SHA-1 response from the server
* @param header
* @param random Random generator of 16 byte websocket keys
* @return
*/
def createRequestHeaders(header:HttpRequestHeader, random: => ByteVector = randomBytes(16)): (HttpRequestHeader, ByteVector) = {
val key = random
val headers =
header.headers.filterNot ( h =>
h.isInstanceOf[`Sec-WebSocket-Key`]
|| h.isInstanceOf[`Sec-WebSocket-Version`]
|| h.isInstanceOf[Upgrade]
) ++
List(
`Sec-WebSocket-Key`(key)
, `Sec-WebSocket-Version`(13)
, Connection(List("upgrade"))
, Upgrade(List(ProductDescription("websocket", None)))
)
header.copy(
method = HttpMethod.GET
, headers = headers
) -> computeFingerPrint(key)
}
/** random generator, ascii compatible **/
def randomBytes(size: Int):ByteVector = {
ByteVector.view(Random.alphanumeric.take(size).mkString.getBytes)
}
/**
* Validates response received. If other than 101 status code is received, this evaluates to Some()
* If fingerprint won't match or the websocket headers wont match the request, this fails.
* @param request Sent request header
* @param response received header
* @param expectFingerPrint expected fingerprint in header
* @return
*/
def validateResponse[F[_] : RaiseThrowable](
request: HttpRequestHeader
, response: HttpResponseHeader
, expectFingerPrint: ByteVector
): Stream[F, Option[HttpResponseHeader]] = {
import Stream._
def validateFingerPrint: Stream[F,Unit] =
response.headers.collectFirst {
case `Sec-WebSocket-Accept`(receivedFp) =>
if (receivedFp != expectFingerPrint) raiseError(new Throwable(s"Websocket fingerprints won't match, expected $expectFingerPrint, but got $receivedFp"))
else emit(())
}.getOrElse(raiseError(new Throwable(s"Websocket response is missing the `Sec-WebSocket-Accept` header : $response")))
def validateUpgrade: Stream[F,Unit] =
response.headers.collectFirst {
case Upgrade(pds) if pds.exists { pd => pd.name.equalsIgnoreCase("websocket") && pd.comment.isEmpty } => emit(())
}.getOrElse(raiseError(new Throwable(s"WebSocket response must contain header 'Upgrade: websocket' : $response")))
def validateConnection: Stream[F,Unit] =
response.headers.collectFirst {
case Connection(ids) if ids.exists(_.equalsIgnoreCase("upgrade")) => emit(())
}.getOrElse(raiseError(new Throwable(s"WebSocket response must contain header 'Connection: Upgrade' : $response")))
def validateProtocols: Stream[F,Unit] = {
val received =
response.headers.collectFirst {
case `Sec-WebSocket-Protocol`(protocols) => protocols
}.getOrElse(Nil)
val expected =
request.headers.collectFirst {
case `Sec-WebSocket-Protocol`(protocols) => protocols
}.getOrElse(Nil)
if (expected.diff(received).nonEmpty) raiseError(new Throwable(s"Websocket protocols do not match. Expected $expected, received: $received"))
else emit(())
}
def validateExtensions: Stream[F,Unit] = {
val received =
response.headers.collectFirst {
case `Sec-WebSocket-Extensions`(extensions) => extensions
}.getOrElse(Nil)
val expected =
request.headers.collectFirst {
case `Sec-WebSocket-Extensions`(extensions) => extensions
}.getOrElse(Nil)
if (expected.diff(received).nonEmpty) raiseError(new Throwable(s"Websocket extensions do not match. Expected $expected, received: $received"))
else emit(())
}
if (response.status != HttpStatusCode.SwitchingProtocols) emit(Some(response))
else {
for {
_ <- validateUpgrade
_ <- validateConnection
_ <- validateFingerPrint
_ <- validateProtocols
_ <- validateExtensions
} yield None: Option[HttpResponseHeader]
}
}
}
}
================================================
FILE: src/main/scala/spinoco/fs2/http/websocket/WebSocketRequest.scala
================================================
package spinoco.fs2.http.websocket
import spinoco.protocol.http.Uri.QueryParameter
import spinoco.protocol.http.header.Host
import spinoco.protocol.http.{HostPort, HttpMethod, HttpRequestHeader, Uri}
/**
* Request to establish websocket connection from the client
* @param hostPort Host (port) of the server
* @param header Any Header information. Note that Method will be always GET replacing any other method configured.
* Also any WebSocket Handshake headers will be overriden.
* @param secure True, if the connection shall be secure (wss)
*/
case class WebSocketRequest(
hostPort: HostPort
, header: HttpRequestHeader
, secure: Boolean
)
object WebSocketRequest {
def ws(host: String, port: Int, path: String, params: QueryParameter *): WebSocketRequest = {
val hostPort = HostPort(host, Some(port))
WebSocketRequest(
hostPort = hostPort
, header = HttpRequestHeader(
method = HttpMethod.GET
, path = Uri.Path.fromUtf8String(path)
, headers = List(
Host(hostPort)
)
, query = Uri.Query(params.toList)
)
, secure = false
)
}
def ws(host: String, path: String, params: QueryParameter *): WebSocketRequest =
ws(host, 80, path, params:_*)
def wss(host: String, port: Int, path: String, params: QueryParameter *): WebSocketRequest = {
ws(host, port, path, params:_*).copy(secure = true)
}
def wss(host: String, path: String, params: QueryParameter *): WebSocketRequest =
wss(host, 443, path, params:_*)
}
================================================
FILE: src/main/scala/spinoco/fs2/http/websocket/package.scala
================================================
package spinoco.fs2.http
import cats.effect.{Concurrent, Timer}
import fs2._
import scodec.{Decoder, Encoder}
import spinoco.protocol.http.HttpRequestHeader
import scala.concurrent.duration._
package object websocket {
/**
* Creates a websocket to be used on server side.
*
* Implementation is according to RFC-6455 (https://tools.ietf.org/html/rfc6455).
*
* @param pipe A websocket pipe. `I` is received from the client and `O` is sent to client.
* Decoder (for I) and Encoder (for O) must be supplied.
* Note that this function may evaluate on the left, to indicate response to the client before
* the handshake took place (i.e. Unauthorized).
* @param pingInterval An interval for the Ping / Pong protocol.
* @param handshakeTimeout An timeout to await for handshake to be successfull. If the handshake is not completed
* within supplied period, connection is terminated.
* @tparam F
* @return
*/
def server[F[_] : Concurrent : Timer, I : Decoder, O : Encoder](
pipe: Pipe[F, Frame[I], Frame[O]]
, pingInterval: Duration = 30.seconds
, handshakeTimeout: FiniteDuration = 10.seconds
)(header: HttpRequestHeader, input: Stream[F,Byte]): Stream[F, HttpResponse[F]] =
WebSocket.server(pipe, pingInterval, handshakeTimeout)(header, input)
}
================================================
FILE: src/test/scala/spinoco/fs2/http/HttpRequestSpec.scala
================================================
package spinoco.fs2.http
import cats.effect.IO
import fs2._
import org.scalacheck.Properties
import org.scalacheck.Prop._
import spinoco.protocol.http._
import spinoco.protocol.http.codec.HttpRequestHeaderCodec
import spinoco.protocol.http.header._
import spinoco.protocol.mime.{ContentType, MIMECharset, MediaType}
object HttpRequestSpec extends Properties("HttpRequest") {
import spinoco.fs2.http.util.chunk2ByteVector
property("encode") = secure {
val request =
HttpRequest.get[IO](
Uri.http("www.spinoco.com", "/hello-world.html")
).withUtf8Body("Hello World")
HttpRequest.toStream(request, HttpRequestHeaderCodec.defaultCodec)
.chunks.compile.toVector.map { _.map(chunk2ByteVector).reduce { _ ++ _ }.decodeUtf8 }
.unsafeRunSync() ?=
Right(Seq(
"GET /hello-world.html HTTP/1.1"
, "Host: www.spinoco.com"
, "Content-Type: text/plain; charset=utf-8"
, "Content-Length: 11"
, ""
, "Hello World"
).mkString("\r\n"))
}
property("decode") = secure {
Stream.chunk(Chunk.bytes(
Seq(
"GET /hello-world.html HTTP/1.1"
, "Host: www.spinoco.com"
, "Content-Type: text/plain; charset=utf-8"
, "Content-Length: 11"
, ""
, "Hello World"
).mkString("\r\n").getBytes
))
.covary[IO]
.through(HttpRequest.fromStream[IO](4096,HttpRequestHeaderCodec.defaultCodec))
.flatMap { case (header, body) =>
Stream.eval(body.chunks.compile.toVector.map(_.map(chunk2ByteVector).reduce(_ ++ _).decodeUtf8)).map { bodyString =>
header -> bodyString
}
}.compile.toVector.unsafeRunSync() ?= Vector(
HttpRequestHeader(
method = HttpMethod.GET
, path = Uri.Path / "hello-world.html"
, headers = List(
Host(HostPort("www.spinoco.com", None))
, `Content-Type`(ContentType.TextContent(MediaType.`text/plain`, Some(MIMECharset.`UTF-8`)))
, `Content-Length`(11)
)
, query = Uri.Query.empty
) -> Right("Hello World")
)
}
}
================================================
FILE: src/test/scala/spinoco/fs2/http/HttpResponseSpec.scala
================================================
package spinoco.fs2.http
import cats.effect.IO
import fs2._
import org.scalacheck.Properties
import org.scalacheck.Prop._
import scodec.Attempt
import spinoco.protocol.http.header._
import spinoco.protocol.http.codec.HttpResponseHeaderCodec
import spinoco.protocol.http.{HttpResponseHeader, HttpStatusCode}
import spinoco.fs2.http.util.chunk2ByteVector
import spinoco.protocol.mime.{ContentType, MIMECharset, MediaType}
object HttpResponseSpec extends Properties("HttpResponse") {
property("encode") = secure {
val response =
HttpResponse[IO](HttpStatusCode.Ok)
.withUtf8Body("Hello World")
HttpResponse.toStream(response, HttpResponseHeaderCodec.defaultCodec)
.chunks.compile.toVector.map { _.map(chunk2ByteVector).reduce { _ ++ _ }.decodeUtf8 }
.unsafeRunSync() ?=
Right(Seq(
"HTTP/1.1 200 OK"
, "Content-Type: text/plain; charset=utf-8"
, "Content-Length: 11"
, ""
, "Hello World"
).mkString("\r\n"))
}
property("decode") = secure {
Stream.chunk(Chunk.bytes(
Seq(
"HTTP/1.1 200 OK"
, "Content-Type: text/plain; charset=utf-8"
, "Content-Length: 11"
, ""
, "Hello World"
).mkString("\r\n").getBytes
))
.covary[IO]
.through(HttpResponse.fromStream[IO](4096, HttpResponseHeaderCodec.defaultCodec))
.flatMap { response => Stream.eval(response.bodyAsString).map(response.header -> _ ) }
.compile.toVector.unsafeRunSync() ?=
Vector(
HttpResponseHeader(
status = HttpStatusCode.Ok
, reason = "OK"
, headers = List(
`Content-Type`(ContentType.TextContent(MediaType.`text/plain`, Some(MIMECharset.`UTF-8`)))
, `Content-Length`(11)
)
) -> Attempt.Successful("Hello World")
)
}
}
================================================
FILE: src/test/scala/spinoco/fs2/http/HttpServerSpec.scala
================================================
package spinoco.fs2.http
import java.net.InetSocketAddress
import cats.effect.IO
import fs2._
import org.scalacheck.Properties
import org.scalacheck.Prop._
import spinoco.fs2.http
import spinoco.fs2.http.body.BodyEncoder
import spinoco.protocol.http.header.{`Content-Length`, `Content-Type`}
import spinoco.protocol.mime.{ContentType, MediaType}
import spinoco.protocol.http.{HttpRequestHeader, HttpStatusCode, Uri}
import scala.concurrent.duration._
object HttpServerSpec extends Properties("HttpServer"){
import Resources._
val MaxConcurrency: Int = 10
def echoService(request: HttpRequestHeader, body: Stream[IO,Byte]): Stream[IO,HttpResponse[IO]] = {
if (request.path != Uri.Path / "echo") Stream.emit(HttpResponse[IO](HttpStatusCode.Ok).withUtf8Body("Hello World")).covary[IO]
else {
val ct = request.headers.collectFirst { case `Content-Type`(ct0) => ct0 }.getOrElse(ContentType.BinaryContent(MediaType.`application/octet-stream`, None))
val size = request.headers.collectFirst { case `Content-Length`(sz) => sz }.getOrElse(0l)
val ok = HttpResponse(HttpStatusCode.Ok).chunkedEncoding.withContentType(ct).withBodySize(size)
Stream.emit(ok.copy(body = body.take(size)))
}
}
def failRouteService(request: HttpRequestHeader, body: Stream[IO,Byte]): Stream[IO,HttpResponse[IO]] = {
Stream.raiseError(new Throwable("Booom!"))
}
def failingResponse(request: HttpRequestHeader, body: Stream[IO,Byte]): Stream[IO,HttpResponse[IO]] = Stream {
HttpResponse(HttpStatusCode.Ok).copy(body = Stream.raiseError(new Throwable("Kaboom!")).covary[IO])
}.covary[IO]
property("simultaneous-requests") = secure {
// run up to count parallel requests and then make sure all of them pass within timeout
val count = 100
def clients : Stream[IO, Stream[IO, (Int, Boolean)]] = {
val request = HttpRequest.get[IO](Uri.parse("http://127.0.0.1:9090/echo").getOrElse(throw new Throwable("Invalid uri")))
Stream.eval(client[IO]()).flatMap { httpClient =>
Stream.range(0,count).unchunk.map { idx =>
httpClient.request(request).map(resp => idx -> (resp.header.status == HttpStatusCode.Ok))
}}
}
(Stream(
http.server[IO](new InetSocketAddress("127.0.0.1", 9090))(echoService).drain
).covary[IO] ++ Stream.sleep_[IO](1.second) ++ clients)
.parJoin(MaxConcurrency)
.take(count)
.filter { case (idx, success) => success }
.compile.toVector.unsafeRunTimed(30.seconds).map { _.size } ?= Some(count)
}
property("simultaneous-requests-echo body") = secure {
// run up to count parallel requests with body, and then make sure all of them pass within timeout with body echoed back
val count = 100
def clients : Stream[IO, Stream[IO, (Int, Boolean)]] = {
val request =
HttpRequest.get[IO](Uri.parse("http://127.0.0.1:9090/echo").getOrElse(throw new Throwable("Invalid uri")))
.withBody("Hello")(BodyEncoder.utf8String, RaiseThrowable.fromApplicativeError[IO])
Stream.eval(client[IO]()).flatMap { httpClient =>
Stream.range(0,count).unchunk.map { idx =>
httpClient.request(request).flatMap { resp =>
Stream.eval(resp.bodyAsString).map { attempt =>
val okResult = resp.header.status == HttpStatusCode.Ok
attempt.map(_ == "Hello").map(r => idx -> (r && okResult)).getOrElse(idx -> false)
}
}
}}
}
( Stream.sleep_[IO](3.second) ++
(Stream(
http.server[IO](new InetSocketAddress("127.0.0.1", 9090))(echoService).drain
).covary[IO] ++ Stream.sleep_[IO](3.second) ++ clients).parJoin(MaxConcurrency))
.take(count)
.filter { case (idx, success) => success }
.compile.toVector.unsafeRunTimed(60.seconds).map { _.size } ?= Some(count)
}
property("request-failed-to-route") = secure {
// run up to count parallel requests with body, server shall fail each, nevertheless response shall be delivered.
val count = 1
def clients : Stream[IO, Stream[IO, (Int, Boolean)]] = {
val request =
HttpRequest.get[IO](Uri.parse("http://127.0.0.1:9090/echo").getOrElse(throw new Throwable("Invalid uri")))
Stream.eval(client[IO]()).flatMap { httpClient =>
Stream.range(0,count).unchunk.map { idx =>
httpClient.request(request).map { resp =>
idx -> (resp.header.status == HttpStatusCode.BadRequest)
}}}
}
(Stream.sleep_[IO](3.second) ++
(Stream(
HttpServer[IO](
bindTo = new InetSocketAddress("127.0.0.1", 9090)
, service = failRouteService
, requestFailure = _ => { Stream(HttpResponse[IO](HttpStatusCode.BadRequest)).covary[IO] }
, sendFailure = HttpServer.handleSendFailure[IO] _
).drain
).covary[IO] ++ Stream.sleep_[IO](1.second) ++ clients).parJoin(MaxConcurrency))
.take(count)
.filter { case (idx, success) => success }
.compile.toVector.unsafeRunTimed(30.seconds).map { _.size } ?= Some(count)
}
property("request-failed-body-send") = secure {
// run up to count parallel requests with body, server shall fail each (when sending body), nevertheless response shall be delivered.
val count = 100
def clients : Stream[IO, Stream[IO, (Int, Boolean)]] = {
val request =
HttpRequest.get[IO](Uri.parse("http://127.0.0.1:9090/echo").getOrElse(throw new Throwable("Invalid uri")))
Stream.eval(client[IO]()).flatMap { httpClient =>
Stream.range(0,count).unchunk.map { idx =>
httpClient.request(request).map { resp =>
idx -> (resp.header.status == HttpStatusCode.Ok) // body won't be consumed, and request was succesfully sent
}
}
}
}
(Stream.sleep_[IO](3.second) ++
(Stream(
HttpServer[IO](
bindTo = new InetSocketAddress("127.0.0.1", 9090)
, service = failingResponse
, requestFailure = HttpServer.handleRequestParseError[IO] _
, sendFailure = (_, _, _) => Stream.empty
).drain
).covary[IO] ++ Stream.sleep_[IO](1.second) ++ clients).parJoin(MaxConcurrency))
.take(count)
.filter { case (idx, success) => success }
.compile.toVector.unsafeRunTimed(30.seconds).map { _.size } ?= Some(count)
}
}
================================================
FILE: src/test/scala/spinoco/fs2/http/Resources.scala
================================================
package spinoco.fs2.http
import java.nio.channels.AsynchronousChannelGroup
import java.util.concurrent.Executors
import cats.effect.{Concurrent, ContextShift, IO, Timer}
import scala.concurrent.ExecutionContext
object Resources {
implicit val _cxs: ContextShift[IO] = IO.contextShift(ExecutionContext.Implicits.global)
implicit val _timer: Timer[IO] = IO.timer(ExecutionContext.Implicits.global)
implicit val _concurrent: Concurrent[IO] = IO.ioConcurrentEffect(_cxs)
implicit val AG: AsynchronousChannelGroup = AsynchronousChannelGroup.withThreadPool(Executors.newCachedThreadPool(util.mkThreadFactory("fs2-http-spec-AG", daemon = true)))
}
================================================
FILE: src/test/scala/spinoco/fs2/http/internal/ChunkedEncodingSpec.scala
================================================
package spinoco.fs2.http.internal
import cats.effect.IO
import fs2._
import org.scalacheck.Properties
import org.scalacheck.Prop._
import scodec.bits.ByteVector
import spinoco.fs2.http.util.chunk2ByteVector
object ChunkedEncodingSpec extends Properties("ChunkedEncoding") {
property("encode-decode") = forAll { strings: List[String] =>
val in = strings.foldLeft(Stream.empty.covaryAll[IO, Byte]) { case(s,n) => s ++ Stream.chunk(Chunk.bytes(n.getBytes)) }
(in through ChunkedEncoding.encode through ChunkedEncoding.decode(1024))
.chunks
.compile.toVector
.map(_.foldLeft(ByteVector.empty){ case (bv, n) => bv ++ chunk2ByteVector(n) })
.map(_.decodeUtf8)
.unsafeRunSync() ?= Right(
strings.mkString
)
}
val wikiExample = "4\r\n" +
"Wiki\r\n" +
"5\r\n" +
"pedia\r\n" +
"E\r\n" +
" in\r\n"+
"\r\n" +
"chunks.\r\n" +
"0\r\n" +
"\r\n"
property("encoded-wiki-example") = secure {
(Stream.chunk[IO, Byte](Chunk.bytes(wikiExample.getBytes)) through ChunkedEncoding.decode(1024))
.covary[IO]
.chunks
.compile.toVector
.map(_.foldLeft(ByteVector.empty){ case (bv, n) => bv ++ chunk2ByteVector(n) })
.map(_.decodeUtf8)
.unsafeRunSync() ?= Right(
"Wikipedia in\r\n\r\nchunks."
)
}
property("decoded-wiki-example") = secure {
val chunks:Stream[IO,Byte] = Stream.emits(
Seq(
"Wiki"
, "pedia"
, " in\r\n\r\nchunks."
)
).flatMap(s => Stream.chunk[IO, Byte](Chunk.bytes(s.getBytes)))
(chunks through ChunkedEncoding.encode)
.chunks
.compile.toVector
.map(_.foldLeft(ByteVector.empty){ case (bv, n) => bv ++ chunk2ByteVector(n) })
.map(_.decodeUtf8)
.unsafeRunSync() ?= Right(wikiExample)
}
}
================================================
FILE: src/test/scala/spinoco/fs2/http/internal/HttpClientApp.scala
================================================
package spinoco.fs2.http.internal
import cats.effect.IO
import fs2._
import spinoco.fs2.http
import spinoco.fs2.http.HttpRequest
import spinoco.protocol.http.Uri
object HttpClientApp extends App {
import spinoco.fs2.http.Resources._
http.client[IO]().flatMap { httpClient =>
httpClient.request(HttpRequest.get(Uri.https("www.google.cz", "/"))).flatMap { resp =>
Stream.eval(resp.bodyAsString)
}.compile.toVector.map {
println
}
}.unsafeRunSync()
}
================================================
FILE: src/test/scala/spinoco/fs2/http/internal/HttpServerApp.scala
================================================
package spinoco.fs2.http.internal
import java.net.InetSocketAddress
import cats.effect.IO
import fs2._
import spinoco.fs2.http
import spinoco.fs2.http.HttpResponse
import spinoco.protocol.http.header._
import spinoco.protocol.mime.{ContentType, MediaType}
import spinoco.protocol.http.{HttpRequestHeader, HttpStatusCode, Uri}
object HttpServerApp extends App {
import spinoco.fs2.http.Resources._
def service(request: HttpRequestHeader, body: Stream[IO,Byte]): Stream[IO,HttpResponse[IO]] = {
if (request.path != Uri.Path / "echo") Stream.emit(HttpResponse[IO](HttpStatusCode.Ok).withUtf8Body("Hello World")).covary[IO]
else {
val ct = request.headers.collectFirst { case `Content-Type`(ct) => ct }.getOrElse(ContentType.BinaryContent(MediaType.`application/octet-stream`, None))
val size = request.headers.collectFirst { case `Content-Length`(sz) => sz }.getOrElse(0l)
val ok = HttpResponse(HttpStatusCode.Ok).chunkedEncoding.withContentType(ct).withBodySize(size)
Stream.emit(ok.copy(body = body.take(size)))
}
}
http.server(new InetSocketAddress("127.0.0.1", 9090))(service).compile.drain.unsafeRunSync()
}
================================================
FILE: src/test/scala/spinoco/fs2/http/routing/MatcherSpec.scala
================================================
package spinoco.fs2.http.routing
import cats.effect.IO
import fs2._
import org.scalacheck.Properties
import org.scalacheck.Prop._
import spinoco.fs2.http.HttpResponse
import spinoco.protocol.http.{HttpMethod, HttpRequestHeader, HttpStatusCode, Uri}
object MatcherSpec extends Properties("Matcher"){
val request = HttpRequestHeader(
method = HttpMethod.GET
)
def requestAt(s: String): HttpRequestHeader =
request.copy(path = Uri.Path.fromUtf8String(s))
implicit class RouteTaskInstance(val self: Route[IO]) {
def matches(request: HttpRequestHeader, body: Stream[IO, Byte] = Stream.empty.covaryAll[IO, Byte]): MatchResult[IO, Stream[IO, HttpResponse[IO]]] =
Matcher.run(self)(request, body).unsafeRunSync()
}
val RespondOk: Stream[IO, HttpResponse[IO]] = Stream.emit(HttpResponse[IO](HttpStatusCode.Ok)).covary[IO]
property("matches-uri") = secure {
val r: Route[IO] = "hello" / "world" map { _ => RespondOk }
(r matches requestAt("/hello/world") isSuccess) &&
(r matches requestAt("/hello/world2") isFailure)
}
property("matches-uri-alternative") = secure {
val r: Route[IO] = "hello" / choice(
"world"
, "nation" / ( "greeks" or "romans" )
, "city" / "of" / "prague"
) map { _ => RespondOk }
(r matches requestAt("/hello/world") isSuccess) &&
(r matches requestAt("/hello/nation/greeks") isSuccess) &&
(r matches requestAt("/hello/nation/romans") isSuccess) &&
(r matches requestAt("/hello/city/of/prague") isSuccess) &&
(r matches requestAt("/bye") isFailure) &&
(r matches requestAt("/hello/town") isFailure) &&
(r matches requestAt("/hello/nation/egyptians") isFailure) &&
(r matches requestAt("/hello/city/of/berlin") isFailure)
}
property("matches-deep-uri") = secure {
val r: Route[IO] = (1 until 10000).foldLeft[Matcher[IO, String]]("deep" / "0") {
case (m, next) => m / next.toString
} map { _ => RespondOk }
val path = (1 until 10000).mkString("/deep/0/", "/", "")
r matches requestAt(path) isSuccess
}
property("matcher-hlist") = secure {
val r: Route[IO] = "hello" :/: "body" :/: as[Int] :/: "foo" :/: as[Long] map { _ => RespondOk }
r matches(requestAt("/hello/body/33/foo/22")) isSuccess
}
property("matcher-advance-recover") = secure {
val r: Route[IO] = choice(
"hello" / choice ( "user" / "one" ) map { _ => RespondOk }
, "hello" / "people" map { _ => RespondOk }
)
(r matches(requestAt("/hello/people")) isSuccess)
}
property("matches-uri-alternative") = secure {
val r: Route[IO] = "hello" / choice(
"world"
, "nation" / ( "greeks" or "romans" )
, "city" / "of" / "prague"
) map { _ => RespondOk }
(r matches requestAt("/hello/world") isSuccess) &&
(r matches requestAt("/hello/nation/greeks") isSuccess) &&
(r matches requestAt("/hello/nation/romans") isSuccess) &&
(r matches requestAt("/hello/city/of/prague") isSuccess) &&
(r matches requestAt("/bye") isFailure) &&
(r matches requestAt("/hello/town") isFailure) &&
(r matches requestAt("/hello/nation/egyptians") isFailure) &&
(r matches requestAt("/hello/city/of/berlin") isFailure)
}
property("matches-deep-uri") = secure {
val r: Route[IO] = (1 until 10000).foldLeft[Matcher[IO, String]]("deep" / "0") {
case (m, next) => m / next.toString
} map { _ => RespondOk }
val path = (1 until 10000).mkString("/deep/0/", "/", "")
r matches requestAt(path) isSuccess
}
property("matcher-hlist") = secure {
val r: Route[IO] = "hello" :/: "body" :/: as[Int] :/: "foo" :/: as[Long] map { _ => RespondOk }
r matches(requestAt("/hello/body/33/foo/22")) isSuccess
}
}
================================================
FILE: src/test/scala/spinoco/fs2/http/sse/SSEEncodingSpec.scala
================================================
package spinoco.fs2.http.sse
import cats.effect.IO
import fs2.Chunk.ByteVectorChunk
import fs2._
import org.scalacheck.Properties
import org.scalacheck.Prop._
import scodec.bits.ByteVector
import spinoco.fs2.http.sse.SSEMessage.SSEData
import spinoco.fs2.http.util.chunk2ByteVector
object SSEEncodingSpec extends Properties("SSEEncoding") {
property("encode") = secure {
Stream(
SSEMessage.SSEData(Seq("data1"), None, None)
, SSEMessage.SSEData(Seq("data2", "data3"), None, None)
, SSEMessage.SSEData(Seq("data4"), Some("event1"), None)
, SSEMessage.SSEData(Seq("data5"), None, Some("id1"))
, SSEMessage.SSEData(Seq("data6"), Some("event2"), Some("id2"))
).covary[IO].through(SSEEncoding.encode[IO]).chunks.compile.toVector.map { _ map chunk2ByteVector reduce (_ ++ _) decodeUtf8 }.unsafeRunSync() ?=
Right(
"data: data1\n\ndata: data2\ndata: data3\n\nevent: event1\ndata: data4\n\ndata: data5\nid: id1\n\nevent: event2\ndata: data6\nid: id2\n\n"
)
}
property("decode.example.1") = secure {
Stream.chunk(ByteVectorChunk(ByteVector.view(
": test stream\n\ndata: first event\nid: 1\n\ndata:second event\nid\n\ndata: third event".getBytes()
)))
.covary[IO]
.through(SSEEncoding.decode[IO]).compile.toVector.unsafeRunSync() ?=
Vector(
SSEData(Vector("first event"), None, Some("1"))
, SSEData(Vector("second event"), None, Some(""))
)
}
property("decode.example.2") = secure {
Stream.chunk(ByteVectorChunk(ByteVector.view(
"data\n\ndata\ndata\n\ndata:".getBytes()
)))
.covary[IO]
.through(SSEEncoding.decode[IO]).compile.toVector.unsafeRunSync() ?=
Vector(
SSEData(Vector(""), None, None)
, SSEData(Vector("",""), None, None)
)
}
property("decode.example.3") = secure {
Stream.chunk(ByteVectorChunk(ByteVector.view(
"data:test\n\ndata: test\n\n".getBytes()
)))
.covary[IO]
.through(SSEEncoding.decode[IO]).compile.toVector.unsafeRunSync() ?=
Vector(
SSEData(Vector("test"), None, None)
, SSEData(Vector("test"), None, None)
)
}
}
================================================
FILE: src/test/scala/spinoco/fs2/http/util/UtilSpec.scala
================================================
package spinoco.fs2.http.util
import cats.effect.IO
import fs2._
import org.scalacheck.Prop._
import org.scalacheck.{Arbitrary, Gen, Properties}
import scodec.bits.Bases.{Alphabets, Base64Alphabet}
import scodec.bits.ByteVector
import shapeless.the
import spinoco.fs2.http.util
object UtilSpec extends Properties("util"){
case class EncodingSample(chunkSize:Int, text:String, alphabet: Base64Alphabet)
implicit val encodingTestInstance : Arbitrary[EncodingSample] = Arbitrary {
for {
s <- the[Arbitrary[String]].arbitrary
chunkSize <- Gen.choose(1,s.length max 1)
alphabet <- Gen.oneOf(Seq(Alphabets.Base64Url, Alphabets.Base64))
} yield EncodingSample(chunkSize, s, alphabet)
}
property("encodes.base64") = forAll { sample: EncodingSample =>
Stream.chunk[IO, Byte](Chunk.bytes(sample.text.getBytes)).chunkLimit(sample.chunkSize).flatMap(Stream.chunk[IO, Byte])
.through(util.encodeBase64Raw(sample.alphabet))
.chunks
.fold(ByteVector.empty){ case (acc, n) => acc ++ chunk2ByteVector(n)}
.map(_.decodeUtf8)
.compile.toVector.unsafeRunSync() ?= Vector(
Right(ByteVector.view(sample.text.getBytes).toBase64(sample.alphabet))
)
}
property("decodes.base64") = forAll { sample: EncodingSample =>
val encoded = ByteVector.view(sample.text.getBytes).toBase64(sample.alphabet)
Stream.chunk[IO, Byte](Chunk.bytes(encoded.getBytes))
.chunkLimit(sample.chunkSize).flatMap(Stream.chunk[IO, Byte])
.through(util.decodeBase64Raw(sample.alphabet))
.chunks
.fold(ByteVector.empty){ case (acc, n) => acc ++ chunk2ByteVector(n)}
.map(_.decodeUtf8)
.compile.toVector.unsafeRunSync() ?= Vector(
Right(sample.text)
)
}
property("encodes.decodes.base64") = forAll { sample: EncodingSample =>
val r =
Stream.chunk[IO, Byte](Chunk.bytes(sample.text.getBytes)).covary[IO].chunkLimit(sample.chunkSize).flatMap(Stream.chunk[IO, Byte])
.through(util.encodeBase64Raw(sample.alphabet))
.through(util.decodeBase64Raw(sample.alphabet))
.chunks
.fold(ByteVector.empty){ case (acc, n) => acc ++ chunk2ByteVector(n)}
.compile.toVector.unsafeRunSync()
r ?= Vector(ByteVector.view(sample.text.getBytes))
}
}
================================================
FILE: src/test/scala/spinoco/fs2/http/websocket/WebSocketClientApp.scala
================================================
package spinoco.fs2.http.websocket
import cats.effect.IO
import scala.concurrent.duration._
import fs2._
import scodec.Codec
import scodec.codecs._
import spinoco.protocol.http.Uri.QueryParameter
object WebSocketClientApp extends App {
import spinoco.fs2.http.Resources._
def wspipe: Pipe[IO, Frame[String], Frame[String]] = { inbound =>
val output = Stream.awakeEvery[IO](1.second).map { dur => println(s"SENT $dur"); Frame.Text(s" ECHO $dur") }.take(5)
output concurrently inbound.take(5).map { in => println(("RECEIVED ", in)) }
}
implicit val codecString: Codec[String] = utf8
WebSocket.client(
WebSocketRequest.ws("echo.websocket.org", "/", QueryParameter.single("encoding", "text"))
, wspipe
).map { x =>
println(("RESULT OF WS", x))
}.compile.drain.unsafeRunSync()
}
================================================
FILE: src/test/scala/spinoco/fs2/http/websocket/WebSocketSpec.scala
================================================
package spinoco.fs2.http.websocket
import java.net.InetSocketAddress
import cats.effect.IO
import fs2._
import org.scalacheck.{Gen, Prop, Properties}
import org.scalacheck.Prop._
import scodec.Codec
import scodec.bits.ByteVector
import scodec.codecs._
import spinoco.fs2.http
import scala.concurrent.duration._
object WebSocketSpec extends Properties("WebSocket") {
import spinoco.fs2.http.Resources._
property("random-bytes-size") = {
val interval = Gen.choose(1,50)
Prop.forAll(interval) { size: Int =>
WebSocket.impl.randomBytes(size).length == size
}
}
property("computes-fingerprint") = secure {
val key = ByteVector.fromBase64("L54CF9+DxAZSOHDW3AoG1A==").get
val fp = WebSocket.impl.computeFingerPrint(key)
fp ?= ByteVector.fromBase64("rsNEx/DEOf5YTl9Jd/jPWeKlKbk=").get
}
property("websocket-server") = secure {
implicit val codecString: Codec[String] = utf8
var received:List[Frame[String]] = Nil
def serverEcho: Pipe[IO, Frame[String], Frame[String]] = { in => in }
def clientData: Pipe[IO, Frame[String], Frame[String]] = { inbound =>
val output = Stream.awakeEvery[IO](1.seconds).map { dur => Frame.Text(s" ECHO $dur") }.take(5)
output merge inbound.take(5).evalMap { in => IO { received = received :+ in }}.drain
}
val serverStream =
http.server[IO](new InetSocketAddress("127.0.0.1", 9090))(
server (
pipe = serverEcho
, pingInterval = 500.millis
, handshakeTimeout = 10.seconds
)
)
val clientStream =
Stream.sleep_[IO](3.seconds) ++
WebSocket.client(
WebSocketRequest.ws("127.0.0.1", 9090, "/")
, clientData
)
val resultClient =
(serverStream.drain mergeHaltBoth clientStream).compile.toVector.unsafeRunTimed(20.seconds)
(resultClient ?= Some(Vector(None))) &&
(received.size ?= 5)
}
property("websocket-cut-frame-125-bytes") = secure{
val data = ByteVector.fromHex("827d" + "aa"*170).get
val cut = WebSocket.impl.cutFrame(data)
cut ?= Some(data.take(127))
}
property("websocket-cut-frame-256-bytes") = secure{
val data = ByteVector.fromHex("827e0100" + "aa"*300).get
val cut = WebSocket.impl.cutFrame(data)
cut ?= Some(data.take(260))
}
property("websocket-cut-frame-65536-bytes") = secure{
val data = ByteVector.fromHex("827f0000000000010000" + "aa"*70000).get
val cut = WebSocket.impl.cutFrame(data)
cut ?= Some(data.take(65546))
}
property("websocket-cut-frame-less-16") = secure{
val data = ByteVector.fromHex("827").get
val cut = WebSocket.impl.cutFrame(data)
cut ?= None
}
property("websocket-cut-frame-not-enough") = secure{
val data = ByteVector.fromHex("827e0100" + "aa"*100).get
val cut = WebSocket.impl.cutFrame(data)
cut ?= None
}
}
================================================
FILE: version.sbt
================================================
version in ThisBuild := "0.4.2-SNAPSHOT"
gitextract_8b82oz4x/ ├── .gitignore ├── .travis.yml ├── LICENSE ├── README.md ├── build.sbt ├── doc/ │ └── custom_codec.md ├── project/ │ ├── build.properties │ └── plugins.sbt ├── sbt ├── src/ │ ├── main/ │ │ └── scala/ │ │ └── spinoco/ │ │ └── fs2/ │ │ └── http/ │ │ ├── HttpClient.scala │ │ ├── HttpRequestOrResponse.scala │ │ ├── HttpServer.scala │ │ ├── body/ │ │ │ ├── BodyDecoder.scala │ │ │ ├── BodyEncoder.scala │ │ │ ├── StreamBodyDecoder.scala │ │ │ └── StreamBodyEncoder.scala │ │ ├── http.scala │ │ ├── internal/ │ │ │ ├── ChunkedEncoding.scala │ │ │ └── internal.scala │ │ ├── routing/ │ │ │ ├── MatchResult.scala │ │ │ ├── Matcher.scala │ │ │ ├── StringDecoder.scala │ │ │ └── routing.scala │ │ ├── sse/ │ │ │ ├── SSEDecoder.scala │ │ │ ├── SSEEncoder.scala │ │ │ ├── SSEEncoding.scala │ │ │ └── SSEMessage.scala │ │ ├── util/ │ │ │ └── util.scala │ │ └── websocket/ │ │ ├── Frame.scala │ │ ├── WebSocket.scala │ │ ├── WebSocketRequest.scala │ │ └── package.scala │ └── test/ │ └── scala/ │ └── spinoco/ │ └── fs2/ │ └── http/ │ ├── HttpRequestSpec.scala │ ├── HttpResponseSpec.scala │ ├── HttpServerSpec.scala │ ├── Resources.scala │ ├── internal/ │ │ ├── ChunkedEncodingSpec.scala │ │ ├── HttpClientApp.scala │ │ └── HttpServerApp.scala │ ├── routing/ │ │ └── MatcherSpec.scala │ ├── sse/ │ │ └── SSEEncodingSpec.scala │ ├── util/ │ │ └── UtilSpec.scala │ └── websocket/ │ ├── WebSocketClientApp.scala │ └── WebSocketSpec.scala └── version.sbt
Condensed preview — 45 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (177K chars).
[
{
"path": ".gitignore",
"chars": 203,
"preview": "*.class\n*.log\n\n# sbt specific\n.cache\n.history\n.lib/\ndist/*\ntarget/\nlib_managed/\nsrc_managed/\nproject/boot/\nproject/plugi"
},
{
"path": ".travis.yml",
"chars": 212,
"preview": "\nlanguage : scala\n\nscala:\n - 2.11.12\n - 2.12.6\n\ncache:\n directories:\n - $HOME/.ivy2\n - $HOME/.sbt\n\njdk:\n - oraclej"
},
{
"path": "LICENSE",
"chars": 1064,
"preview": "MIT License\n\nCopyright (c) 2017 Spinoco\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof"
},
{
"path": "README.md",
"chars": 9720,
"preview": "# fs2-http\n\nMinimalistic yet powerful http client and server with scala fs2 library.\n\n[![Build Status](https://travis-ci"
},
{
"path": "build.sbt",
"chars": 3919,
"preview": "import com.typesafe.sbt.pgp.PgpKeys.publishSigned\n\nval ReleaseTag = \"\"\"^release/([\\d\\.]+a?)$\"\"\".r\n\nlazy val contributors"
},
{
"path": "doc/custom_codec.md",
"chars": 4444,
"preview": "# Using custom codec for http headers and requests. \n\nOcassionally it is required to extends headers supported by fs2-ht"
},
{
"path": "project/build.properties",
"chars": 18,
"preview": "sbt.version=1.1.6\n"
},
{
"path": "project/plugins.sbt",
"chars": 228,
"preview": "addSbtPlugin(\"com.github.tkawachi\" % \"sbt-doctest\" % \"0.8.0\")\naddSbtPlugin(\"com.github.gseitz\" % \"sbt-release\" % \"1.0.9\""
},
{
"path": "sbt",
"chars": 20039,
"preview": "#!/usr/bin/env bash\n#\n# A more capable sbt runner, coincidentally also called sbt.\n# Author: Paul Phillips <paulp@improv"
},
{
"path": "src/main/scala/spinoco/fs2/http/HttpClient.scala",
"chars": 7796,
"preview": "package spinoco.fs2.http\n\nimport java.nio.channels.AsynchronousChannelGroup\nimport java.util.concurrent.TimeUnit\n\nimport"
},
{
"path": "src/main/scala/spinoco/fs2/http/HttpRequestOrResponse.scala",
"chars": 11535,
"preview": "package spinoco.fs2.http\n\n\nimport cats.effect.Sync\nimport fs2.Chunk.ByteVectorChunk\nimport fs2.{Stream, _}\nimport scodec"
},
{
"path": "src/main/scala/spinoco/fs2/http/HttpServer.scala",
"chars": 4774,
"preview": "package spinoco.fs2.http\n\nimport java.net.InetSocketAddress\nimport java.nio.channels.AsynchronousChannelGroup\n\nimport ca"
},
{
"path": "src/main/scala/spinoco/fs2/http/body/BodyDecoder.scala",
"chars": 1636,
"preview": "package spinoco.fs2.http.body\n\nimport scodec.bits.ByteVector\nimport scodec.{Attempt, Decoder, Err}\nimport spinoco.protoc"
},
{
"path": "src/main/scala/spinoco/fs2/http/body/BodyEncoder.scala",
"chars": 1993,
"preview": "package spinoco.fs2.http.body\n\nimport scodec.bits.ByteVector\nimport scodec.{Attempt, Encoder, Err}\nimport spinoco.protoc"
},
{
"path": "src/main/scala/spinoco/fs2/http/body/StreamBodyDecoder.scala",
"chars": 914,
"preview": "package spinoco.fs2.http.body\n\nimport fs2._\nimport spinoco.protocol.mime.{ContentType, MIMECharset}\nimport spinoco.fs2.h"
},
{
"path": "src/main/scala/spinoco/fs2/http/body/StreamBodyEncoder.scala",
"chars": 2863,
"preview": "package spinoco.fs2.http.body\n\nimport cats.MonadError\nimport fs2.Chunk.ByteVectorChunk\nimport fs2._\nimport scodec.Attemp"
},
{
"path": "src/main/scala/spinoco/fs2/http/http.scala",
"chars": 3249,
"preview": "package spinoco.fs2\n\nimport java.net.InetSocketAddress\nimport java.nio.channels.AsynchronousChannelGroup\nimport java.uti"
},
{
"path": "src/main/scala/spinoco/fs2/http/internal/ChunkedEncoding.scala",
"chars": 3447,
"preview": "package spinoco.fs2.http.internal\n\nimport fs2.Chunk.ByteVectorChunk\nimport fs2._\nimport scodec.bits.ByteVector\n\nimport s"
},
{
"path": "src/main/scala/spinoco/fs2/http/internal/internal.scala",
"chars": 5134,
"preview": "package spinoco.fs2.http\n\nimport java.net.InetSocketAddress\nimport java.util.concurrent.TimeoutException\n\nimport javax.n"
},
{
"path": "src/main/scala/spinoco/fs2/http/routing/MatchResult.scala",
"chars": 1437,
"preview": "package spinoco.fs2.http.routing\n\n\nimport spinoco.fs2.http.HttpResponse\nimport spinoco.protocol.http.HttpStatusCode\n\n\ntr"
},
{
"path": "src/main/scala/spinoco/fs2/http/routing/Matcher.scala",
"chars": 7454,
"preview": "package spinoco.fs2.http.routing\n\nimport cats.effect.Sync\nimport fs2._\nimport shapeless.ops.function.FnToProduct\nimport "
},
{
"path": "src/main/scala/spinoco/fs2/http/routing/StringDecoder.scala",
"chars": 2451,
"preview": "package spinoco.fs2.http.routing\n\nimport scodec.bits.{Bases, ByteVector}\nimport shapeless.tag\nimport shapeless.tag.@@\n\ni"
},
{
"path": "src/main/scala/spinoco/fs2/http/routing/routing.scala",
"chars": 7496,
"preview": "package spinoco.fs2.http\n\nimport cats.effect.{Concurrent, Effect, Timer}\nimport fs2._\nimport scodec.{Attempt, Decoder, E"
},
{
"path": "src/main/scala/spinoco/fs2/http/sse/SSEDecoder.scala",
"chars": 480,
"preview": "package spinoco.fs2.http.sse\n\nimport scodec.Attempt\n\n\nsealed trait SSEDecoder[A] { self =>\n\n def decode(in: SSEMessage)"
},
{
"path": "src/main/scala/spinoco/fs2/http/sse/SSEEncoder.scala",
"chars": 658,
"preview": "package spinoco.fs2.http.sse\n\nimport scodec.Attempt\n\n\nsealed trait SSEEncoder[A] { self =>\n\n def encode(a: A) : Attempt"
},
{
"path": "src/main/scala/spinoco/fs2/http/sse/SSEEncoding.scala",
"chars": 5238,
"preview": "package spinoco.fs2.http.sse\n\nimport fs2.Chunk.ByteVectorChunk\nimport fs2._\nimport scodec.Attempt\nimport scodec.bits.Byt"
},
{
"path": "src/main/scala/spinoco/fs2/http/sse/SSEMessage.scala",
"chars": 634,
"preview": "package spinoco.fs2.http.sse\n\nimport scala.concurrent.duration.FiniteDuration\n\n/**\n * SSE Message modeled after\n * htt"
},
{
"path": "src/main/scala/spinoco/fs2/http/util/util.scala",
"chars": 5874,
"preview": "package spinoco.fs2.http\n\nimport java.lang.Thread.UncaughtExceptionHandler\nimport java.util.concurrent.{Executors, Threa"
},
{
"path": "src/main/scala/spinoco/fs2/http/websocket/Frame.scala",
"chars": 313,
"preview": "package spinoco.fs2.http.websocket\n\n\nsealed trait Frame[A] { self =>\n def a: A\n def isText: Boolean = self match {\n "
},
{
"path": "src/main/scala/spinoco/fs2/http/websocket/WebSocket.scala",
"chars": 23022,
"preview": "package spinoco.fs2.http.websocket\n\n\nimport java.nio.channels.AsynchronousChannelGroup\nimport java.util.concurrent.Execu"
},
{
"path": "src/main/scala/spinoco/fs2/http/websocket/WebSocketRequest.scala",
"chars": 1576,
"preview": "package spinoco.fs2.http.websocket\n\nimport spinoco.protocol.http.Uri.QueryParameter\nimport spinoco.protocol.http.header."
},
{
"path": "src/main/scala/spinoco/fs2/http/websocket/package.scala",
"chars": 1435,
"preview": "package spinoco.fs2.http\n\nimport cats.effect.{Concurrent, Timer}\nimport fs2._\nimport scodec.{Decoder, Encoder}\n\nimport s"
},
{
"path": "src/test/scala/spinoco/fs2/http/HttpRequestSpec.scala",
"chars": 2076,
"preview": "package spinoco.fs2.http\n\nimport cats.effect.IO\nimport fs2._\nimport org.scalacheck.Properties\nimport org.scalacheck.Prop"
},
{
"path": "src/test/scala/spinoco/fs2/http/HttpResponseSpec.scala",
"chars": 1825,
"preview": "package spinoco.fs2.http\n\nimport cats.effect.IO\nimport fs2._\nimport org.scalacheck.Properties\nimport org.scalacheck.Prop"
},
{
"path": "src/test/scala/spinoco/fs2/http/HttpServerSpec.scala",
"chars": 6276,
"preview": "package spinoco.fs2.http\n\nimport java.net.InetSocketAddress\n\nimport cats.effect.IO\nimport fs2._\nimport org.scalacheck.Pr"
},
{
"path": "src/test/scala/spinoco/fs2/http/Resources.scala",
"chars": 657,
"preview": "package spinoco.fs2.http\n\nimport java.nio.channels.AsynchronousChannelGroup\nimport java.util.concurrent.Executors\n\nimpor"
},
{
"path": "src/test/scala/spinoco/fs2/http/internal/ChunkedEncodingSpec.scala",
"chars": 1794,
"preview": "package spinoco.fs2.http.internal\n\nimport cats.effect.IO\nimport fs2._\nimport org.scalacheck.Properties\nimport org.scalac"
},
{
"path": "src/test/scala/spinoco/fs2/http/internal/HttpClientApp.scala",
"chars": 487,
"preview": "package spinoco.fs2.http.internal\n\nimport cats.effect.IO\nimport fs2._\nimport spinoco.fs2.http\nimport spinoco.fs2.http.Ht"
},
{
"path": "src/test/scala/spinoco/fs2/http/internal/HttpServerApp.scala",
"chars": 1164,
"preview": "package spinoco.fs2.http.internal\n\nimport java.net.InetSocketAddress\n\nimport cats.effect.IO\nimport fs2._\nimport spinoco."
},
{
"path": "src/test/scala/spinoco/fs2/http/routing/MatcherSpec.scala",
"chars": 3744,
"preview": "package spinoco.fs2.http.routing\n\nimport cats.effect.IO\nimport fs2._\nimport org.scalacheck.Properties\nimport org.scalach"
},
{
"path": "src/test/scala/spinoco/fs2/http/sse/SSEEncodingSpec.scala",
"chars": 2138,
"preview": "package spinoco.fs2.http.sse\n\nimport cats.effect.IO\nimport fs2.Chunk.ByteVectorChunk\nimport fs2._\nimport org.scalacheck."
},
{
"path": "src/test/scala/spinoco/fs2/http/util/UtilSpec.scala",
"chars": 2253,
"preview": "package spinoco.fs2.http.util\n\nimport cats.effect.IO\nimport fs2._\nimport org.scalacheck.Prop._\nimport org.scalacheck.{Ar"
},
{
"path": "src/test/scala/spinoco/fs2/http/websocket/WebSocketClientApp.scala",
"chars": 820,
"preview": "package spinoco.fs2.http.websocket\n\n\nimport cats.effect.IO\n\nimport scala.concurrent.duration._\nimport fs2._\nimport scode"
},
{
"path": "src/test/scala/spinoco/fs2/http/websocket/WebSocketSpec.scala",
"chars": 2860,
"preview": "package spinoco.fs2.http.websocket\n\nimport java.net.InetSocketAddress\n\nimport cats.effect.IO\nimport fs2._\nimport org.sca"
},
{
"path": "version.sbt",
"chars": 41,
"preview": "version in ThisBuild := \"0.4.2-SNAPSHOT\"\n"
}
]
About this extraction
This page contains the full source code of the Spinoco/fs2-http GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 45 files (163.5 KB), approximately 44.7k tokens. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.